Digital camera with in-camera software for image correction

ABSTRACT

A system is disclosed for the automated correction of optical and digital aberrations in a digital imaging system. The system includes several main parts, including (a) digital filters, (b) hardware modifications, (c) digital system corrections, (d) digital system dynamics and (e) network aspects. The system solves numerous problems in still and video photography that are presented in the digital imaging environment.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application is a Continuation of U.S. patent applicationSer. No. 13/691,805, filed Dec. 2, 2012, which application is aContinuation of U.S. patent application Ser. No. 12/586,221, filed Sep.18, 2009 (now U.S. Pat. No. 8,451,339), which application is aContinuation of U.S. patent application Ser. No. 11/825,521, filed Jul.6, 2007 (now U.S. Pat. No. 7,612,805), which application claims thebenefit of priority under 35 U.S.C. section 119 from U.S. Provisionalpatent application Ser. No. 60/807,065, filed on Jul. 11, 2006, thedisclosures of which are hereby incorporated by reference in theirentirety for all purposes.

FIELD OF THE INVENTION

The present invention pertains to imaging systems. The inventionpresents electronic methods to optimize optical processing, the digitalcapture of light and post-capture image organization of digital datasets. The system may be used in any device that captures, records,replicates, scans, reproduces, accesses, organizes, modifies, edits orstores images digitally. The invention applies to any imaging systemthat includes interaction between optics, digital image sensors,post-capture integrated circuits and digital storage components. Thesystem includes self-organizing apparatus for optical image collection,electronic organization and optimization of digital data usingintegrated circuits and software applications. The system applies toconsumer and professional still and video photography, includingcinematography, to the processing of images with digital sensors, and tocopying and scanning technologies. The system also applies to complexgrids of multiple video cameras and satellites for surveillance andreconnaissance.

BACKGROUND OF THE INVENTION

The history of photography in the twentieth century is a story offinding solutions for optimizing optical problems. The main challengeshave involved improving lens configurations to optimize image qualityfor film capture. As an example of this, lens aberrations have beenreliably corrected by creating aspherical lens elements in wide-anglelenses and by creating apochromatic lens elements in telephoto lenses.The use of lens element coatings has also improved optical quality.Similarly, the evolution of zoom lenses has included improvement inoptical quality via the combining of complex optical elementconfigurations; as the optical quality of zoom lenses improveddramatically, their simplicity and utility led them to dominate lenssales. Complex computer-aided design (CAD) software has been used totest a broad range of possible lens configurations so as to optimize theoptical performance in terms of clarity and contrast of each lens type,as much as optically possible within economic constraints.

In the last twenty-five years, an additional revolution has occurredwith the advent of auto-focus (AF) technology in still photographic andvideo cameras. Invented by Minolta in the early 1980s, AF technology wasa photographic application of technology developed for the U.S.military. The AF system used an infrared light sensor to reflect lightonto an object that allowed a camera's lens to focus on the object byemploying a motor in the camera. Canon eventually developed improvedmethods of auto focus by using electronic means (viz., micro ultrasonicmotors) to increase speed and accuracy. This AF technology wasintegrated with automatic exposure (AE) technology which developedcomplex algorithms in a “program” mode to combine shutter and aperturedata to match each lens with particular subject matter, as well as anaperture-priority exposure mode and a shutter-priority exposure mode. Ofcourse, the photographer could use manual focus and manual exposure atany time, but these improvements increased the creative process and thecompetitive advantages of camera makers and photographers that employedthem. Ultimately, the combination of these developments allowed ordinaryphotographers to achieve high quality standards.

AE was improved by Nikon, particularly with the use of a “3D colormatrix” system, which included a library of pre-programmed image types.The combination of the improved AF and AE subsystems allowed a dramaticsimplification of photographic imaging because the photographer nolonger labored over the time-consuming focus and the exposure variables.Automated film advance, in the form of built in motor drives, increasedthe working speeds of camera operation as well. Nearly all consumer andprofessional cameras became automated to some degree by the mid-1990s,including formats beyond the 35 mm film standard.

In the last decade, a new technology of image stabilization (IS) hasemerged to help correct the problem of vibrations caused camera shakethat lead to image blur. This technology is implemented in lenses by theuse of gyros to reorient the light plane to compensate for camera shake;when combined with the earlier automated camera technologies such as AFand AE, IS further improves the photographic experience, particularlyfor larger lenses.

In addition to these advances in camera automation, technologiesimproved in the category of artificial flash as well. Withmicroprocessors and sensors employed to measure minute variances, thedevelopment of photographic flash systems allowed the photographer tocontrol the lighting in an environment to some degree. Nikon's use ofdistance information in its flash system advanced the state of the artin flash photography. The combination of AF and AE with automated flashmechanisms provided increased efficiency and simplicity in image captureprocesses.

The last several years have witnessed a revolution in digitalphotography. Because of its simplicity, potential quality improvement,immediate feedback and cost savings, digital photography has captured anincreasing market share relative to film-based cameras. Kodak holds anumber of patents involving the charge coupled device (CCD) forconverting and recording light into electronic format. A competingtechnology for digital capture is complementary metal-oxidesemiconductor (CMOS) which, though developed by Fairchild Semiconductorover forty years ago, is predominantly used in photographic cameras byCanon. Although there are trade-offs in the application of CCD and CMOSdigital sensors for image capture, they both find wide acceptance in themarket.

In order to correct for digital artifacts in image capture mechanisms,anti-aliasing filters are placed in front of digital sensors. Despitethis improvement, digital imaging still has some challenges to overcomein competing with the image quality of film.

At the present time, the most recent advances in digital imaging forprofessional still photography have come from Hasselblad, which offers aPhase One camera back with a 39 MP digital sensor from Kodak. Thissystem uses software that automatically corrects for digital capturelimitations to produce a quality image. Their “digital APO correction”(DAC) technology performs an analysis of meta-data to color-correct thedigital capture resulting in moiré-free images.

At the limits of current technology, a Canadian company, Dalsa, hasproduced a 111 MP (10,560 by 10,560 pixels) CCD digital sensor thatmeasures four inches square. This technology must be mated with largeformat-type lenses with large image area, and may be used for satellitesurveillance applications and for other astronomical applications.

With both the larger sensor surface area of a medium format camerasystem and the high-quality fixed focal-length lenses of Zeiss,Schneider and Rodenstock, the quality of even the top optics will be alimiting barrier to advanced digital sensors' ability to perceivemaximum resolution. Without new improvements in optical and digitaltechnologies, further progression of photographic camera systems will belimited.

Photographic Problems

Though every major advance in photography has solved an importantproblem, there are still remaining photographic problems to be solved inorder to meet the goals of optimizing optical imaging quality whileincreasing simplicity and efficiency and lowering cost. Despite theadvent and evolution of digital imaging, a number of problems haveemerged in the digital realm in addition to earlier problems involvingoptics. Nevertheless, an opportunity exists to solve some of theseproblems via digital approaches. These problems are generallycategorized as optical or digital.

Optical Problems

In the case of optics, lens aberrations are characterized according tolens type, with wide-angle lens problems differentiated from telephotolens problems. Some of the problems affecting wide-angle lenses arosefrom the creation of the single lens reflex (SLR) camera. Before theSLR, the rear element of a lens could be placed in a rangefinder toprotrude to a point immediately in front of the film plane in order tocorrect for aberrations. While the advantages of the mirror mechanisminclude ability to see exactly what one is photographing, because themirror of the SLR flips up during exposure, the rear element of the lensmust be placed in front of the mirror's plane of movement. Thismechanical fact limits lens designs in most 35 mm and medium-formatcamera systems and particularly affects wide-angle lens configurations.

It is very difficult to control the five aberrations of Seidel—sphericalaberration, distortion (barrel distortion and pin cushion distortion),comatic aberration, astigmatism and curvature of field. In wide-angleSLR lenses as they are currently designed, these aberrations areparticularly prominent.

For wide-angle lenses, optical vignetting affects peripheralillumination. Though optical vignetting will affect even retrofocuswide-angle lenses in rangefinders, it is particularly prominent in SLRcameras. According to the Cosine law, light fall-off in peripheral areasof an image increases as the angle-of-view increases. While opticalvignetting can be reduced by stopping down the lens, the aberrations inrectilinear wide-angle lenses generally exhibit more distortion thewider the lens.

In the case of wide-angle lenses, the depth of field range is muchbroader, with close focusing causing aberrations without stopping downthe aperture. To solve this problem, close-distance focusing is improvedby the creation of floating groups of lens elements. The rear lens groupelements float to correct close-distance aberrations. With wide-anglelenses that have wide apertures, floating lens elements improve lensaberrations in focusing on distance points also.

Modulation transfer function (MTF) curves represent a quantitativemethodology used to assess the resolution and contrast of lensperformance at specific apertures. Each lens type has a specific lenselement composition, formula and behavior as measured by MTF. Ingeneral, MTF measures lens sharpness to 30 lines/mm and contrast to 10lines/mm.

Because different colors of the visible light spectrum behave uniquely,the goal of lens design is to have all colors accurately hit a filmplane or digital sensor plane. The particular challenge for telephotolenses is that the red and green light colors strike the film plane atdifferent times than blue light colors; thus a compensation must be madein the lens configuration to adjust for chromatic aberrations. Cameralens manufacturers have used extra low dispersion glass and fluoriteglass elements in telephoto lenses primarily to adjust the red colorlight spectra to the film plane. In addition, telephoto lenses usecarefully designed lens coatings to limit light diffraction-basedaberrations.

Due to their construction, super-telephoto lenses are very large andheavy. While modifying the materials used in the lens barrels couldreduce size and weight problems, a technological improvement intelephoto lens design was developed by Canon with the addition ofdiffractive optical (DO) elements, which behave as a sort ofhighly-refined fresnel lens magnifier. Though the MTF analyses ofwide-angle lenses show dramatic latitude in performance of even highquality SLR lenses, with particular loss in resolution and contrasttoward the edges of the image, high quality telephoto lenses showcontrol of aberrations. However, the price of these lenses isprohibitively high.

In the case of zoom lenses, as many as four distinct groups of lenselements correct various optical aberrations. These lens element groupsinclude (a) a focusing group, (b) a magnification variation group, (c) acorrection group and (d) an image formation group. Modulating the focallength range of a zoom lens enables the lens to perform within the scopeof operation, yet the zoom lens architecture has limits. In particular,the zoom lens configuration sacrifices resolution and wide potentialaperture. Generally, the degree of resolution and contrast at thesmaller angle of view is traded away in favor of competence at a widerangle of view, or vice-versa. This explains why MTF analyses of zoomlenses generally show a dramatic lowering in resolution and contrastrelative to excellent fixed focal length lenses.

Digital Problems

Digital photography has built on the edifice of film camera systems. Forinstance, the size of the sensor is generally limited to the size of theoptical circumference of a lens system. In the case of 35 mm lenses thatare designed for a specific angle of view, the largest that a digitalsensor in a 35 mm lens system could be, is 24 mm by 36 mm, with acorresponding maximum image circle of 43 mm. In the case of mediumformat lenses, the largest digital sensors would duplicate thecorresponding film plane size, whether 6×4.5 cm, 6×6 cm, 6×7 cm, 6×8 cm,6×9 cm, 6×12 cm or 6×17 cm (which results in an effective image circleas large as 7 inches).

Digital sensors that are smaller than the limits of a corresponding lenssystem have been introduced. For example, Nikon digital sensors aresmaller than 24 mm×36 mm, or advanced photo system (APS) size. Efficientstacking of pixels allows a smaller sensor to eventually match theperformance of a corresponding film system, while using the smallercircumference of the same lenses. Since the outside edges of the lenstypically degrade resolution and contrast, this model using the smallerdigital sensor can have an advantage of using primarily the centralized“sweet spot” of the image area. However, this smaller sensor sizesacrifices the peripheral effects of a wide-angle lens, so a 14 mmbecomes a 21 mm in a 1.5× conversion-sized sensor in a 35 mm lenssystem. On the other hand, with telephoto lenses, the angle of view islimited to the center 65% of the image. This gives the appearance ofupconverting a telephoto lens by 1.5× and thus provides an impression ofincreased magnification; a 400 mm f/2.8 lens appears as a 600 mm f/2.8lens on a camera with a cropped digital sensor. Ultra-wide-angle lenseshave been introduced with smaller image areas than 35 mm to compensatefor smaller sensor size.

Though invented over thirty years ago by Dr. Bayer, the charge coupleddevice (CCD) that is used in many digital cameras generally emulates thebehavior of film. Specifically, since most photographic film has threelayers of green, red and blue, with green representing fifty percent ofthe emulsion and red and blue twenty-five percent each, the CCDarchitecture also configured pixels to capture fifty percent of thegreen photonic visible light spectrum and twenty-five percent each forpixels recording red and blue light. Human eyes see more green than redand blue, so both film and digital sensors seek to emulate the way thatwe see. Each light color is captured by a different pixel in the CCD,just as there are three emulsion layers of film. In recent years, Foveonhas developed a digital sensor for image capture that further seeks toemulate film by structuring the pixels into three layers, again withfifty percent capturing green light and twenty-five percent eachcapturing red and blue light.

Unfortunately, unwanted artifacts are also captured by the digital imagecapture process. These include banding and moiré effects that presentfalse patterns and colors. Moiré patterns are created because the dotpattern of a sensor will intermittently overlap with the pattern of asubject to create a third pattern; these effects are optically-generateddigital distortions that represent the effect of light hitting a pixelwithout correction. In order to compensate for these effects, digitalsensors have employed low pass filters consisting of liquid crystalstructures; however, these filters tend to have the effect of softeningimage resolution. Additionally, RGB or CMYG color filters are placed infront of digital sensors to ensure the accurate capture of colors.

CMOS digital sensors present an alternative to CCDs. By employingalternating positive and negative transistor networks, the CMOS sensorsuse less power. While they do not have the low noise ratio of the CCD,they do have greater light exposure latitude, in both range of ISO anddynamic range of detail in highlight and shadow. More importantly, CMOSsensors contain the circuitry, including analog to digital converter(ADC) and digital to analog converter (DAC), for post-processing digitalimages on the chip itself and enabling increased micro-miniaturizationof the digital imaging process. An increase in the bit rate of the CMOSchip up to 32-bit makes possible a much richer color palate and level ofdetail than with earlier generation CCDs.

CMOS sensors can be full-frame, matching the lens specifications for thecamera systems for which they are designed. A relatively bigger sensorhas a wider depth of field capability, so the background can appear as ablur to set apart the main subject. Given these capabilities, oneproblem that emerges is that a digital sensor's enhanced capabilities tocapture details may exceed the maximum optical resolution capabilitiesof many lenses. Nevertheless, CMOS sensors still require ananti-aliasing filter to be used in front of the sensor, which marginallydegrades resolution.

Over the years, cameras have gotten smaller. While in the 19th centurycameras were 11×14 or 8×10, literally capturing images on large emulsionplates, cameras of today are smaller and more automated. Yet the largerthe film size, the bigger the enlargement potential and the increase inrelative detail in the overall image. Similarly in digital photography,the larger the sensor, the more detail available and the bigger theoutput print can be enlarged. Because of this correspondence of digitalsensor to film, the evolution of digital photography has been restrictedto respective film camera systems, with 35 mm and medium format systemsdominating the field because well-developed lens systems have alreadybeen organized for these camera formats. The potential exists, however,to develop 35 mm camera system digital sensors that rival film-basedmedium format or large format camera system quality or to surpass thelimits of 35 mm camera systems with medium format camera system digitalsensors. The relative size, cost and automation advantages of 35 mmcamera systems generally show that these systems not only arecompetitive, but that these markets are increasingly acceleratedrelative to larger format systems. For example, the development of largeaperture lenses, super-telephotos, rapid auto-focus, refined automatedexposure and image stabilization systems in 35 mm systems has solvedvarious problems that have emerged in the last century and has improvedimage quality and camera system efficiency.

However, in the digital imaging realm additional problems have emerged,including the need to improve color (hue and saturation) quality,exposure highlight range, contrast range and other tonal adjustments. Inaddition, digital image capture brings its own set of aberrations,including moiré and banding effects and noise and ISO range limits.Additional aberrations are linked to the unique design of each type ofdigital sensor, with trade-offs presented between types of CCDs or CMOSchips. Moreover, there are still optical problems in the digital realm,namely, a range of optical aberrations created particularly bywide-angle and zoom lenses as well as the limits of very large, costlyand heavy super-telephoto lenses.

In order to transcend the optical and digital limits of present camerasystems, software systems have been developed that deal with theproblems in post-production. While the most notable of thesepost-production digital editing software programs is Adobe Photoshop,each camera manufacturer has its own proprietary program. In the main,these post-production software programs are limited to color correctionand sharpening/softening of images. Additionally, some of these softwareprograms are able to emulate specific artificial filter techniques toproduce creative modifications of an original image. Nevertheless,manipulating unfiltered RAW image files in post-production processes istime-consuming and expensive and requires considerable skill.

One unintended effect of using digital sensors to capture images indigital photography is that dust accumulates on the sensor surface andthereby obstructs a clear optical image. The vacuum behavior ofincreasingly ubiquitous zoom lenses moves dust in lenses that are notinternally sealed and the existence of dust is a prevalent feature ofdigital photography. Dust on the sensor is a non-trivial problem thatrequires tedious post-production correction for each image. Theexistence of dust on a digital sensor is an inconvenient impediment toachievement of optical imaging quality.

What is needed to correct these various optical and digital aberrationsand unfiltered image files is in-camera modification capability for eachspecific image problem. The present invention describes a digitalimaging system to optimize optical results.

Applications of the Present Invention

The present invention has several practical applications. The digitalimaging system applies to consumer and professional still and videocamera technologies, to cellular telephones and to personal digitalassistants (PDAs). Since video technologies are evolving into digitalformats, including HDTV and its successors, the present invention wouldbe applicable to these technologies. The present invention also appliesto networks of remote camera sensors for surveillance. The system alsoapplies to native digital cinematography and telecine conversion fromanalogue to digital media. The system may be applied to image scanningand image copying technologies. Finally, the present system may beapplied to any optical digital capture, processing and storagesub-system technologies, including groups of sensors or satellites.

Advantages of the Present Invention

There are several important advantages of the present invention. Thepresent system dramatically improves image quality. This digital imagingsystem eliminates, or minimizes, post-production processes. In addition,the system presents substantial cost savings as well as time savingsrelative to existing digital imaging systems because of the automationof integrated in-camera optical and digital corrections of aberrations.

With the present system, in-camera digital filters will largely replaceoptical filters. With less distortion, cost and weight/size than currentoptical filters and with far more control and quality results, thepresent system demonstrates that in-camera digital filtrationout-performs better than external filters. The in-camera filter systemis also an improvement over post-production digital filter systems whichrequire substantial time, cost and skill to implement.

By using the present invention, users will be able to use lens aperturesthat are wider open, because lens aberrations will be corrected. Inturn, this increased latitude will allow more efficient (2+ stops) useof available light and will allow slower lenses to achieve qualityimages that have traditionally been in the domain of expensive fastlenses. The process of photography will be made generally moreefficient.

The present system allows camera manufacturers to design different,predominantly smaller, lenses. Moreover, with providing a system fordigital corrections, camera lenses may use less expensive optics andstill obtain good results. This process will, in turn, accelerate theadoption of low-cost digital photographic systems and, in the aggregate,increase the overall size of the digital photographic market. Similarly,there will be an increase in the effectiveness of zoom lenses, which arealready becoming popular. With the present system, 35 mm optics mayobtain the quality typically recognized by 4×5 film camera systems. Thepresent system will thus facilitate photographic digital imaging tosupplant the film era.

Its application to digital scanning and copying will allow the presentsystem to become ubiquitous as a system to improve imaging.

Because the system presents an integration of several criticalsubsystems that are centered on digital imaging processes, the systemhas applications to video photography, satellite systems andsurveillance systems. The dynamics of the operation of the subsystems ofthe present digital imaging system reveal the refinement, efficiency andoptimization of the digital photographic paradigm.

Importantly, since the present system uses apparatus and methods thatare implemented with software processes, in-camera software can beconstantly upgraded and remotely downloaded, while image files can beautomatically uploaded, organized and published.

If the goal of the evolution of photographic technology is to increasethe number and quality of excellent images, the present system willrevolutionize photography.

Solutions to Digital Imaging Problems that the Present System Presents

The present system provides in-camera digital corrections to bothoptical and digital aberrations. In the case of optical aberrations,distinct lens-specific imperfections are caused by wide-angle, telephotoand zoom lens types. In the case of digital aberrations, specificdigital artifacts, such as aliasing and dust, must be corrected.

The present invention also designs specific improvements to digitalsensor design. These hardware architecture modifications allow digitalimaging to maximize both optical resolution and image exposure range.

Post-capture in-camera filtration is only part of the solution. There isalso a need to modify the digital sensor, whether CCD or CMOS, withspecific filtration in some cases, including a low pass anti-aliasingfilter and a polarizing filter. This post-optic pre-sensor filtrationworks with post-sensor in-camera filtration processes.

In the context of specific complex processes, such as in zoom lensmodifications of focal length and in full-motion video processing,additional corrections are performed using the digital imaging systemwhich continuously optimizes performance.

Finally, the present system reveals approaches to improve networkcoordination and wireless external storage and publication capabilitiesfor digital imaging systems.

BRIEF SUMMARY OF THE INVENTION

The present invention is divided into several main sub-systems: (1)digital filters to correct optical and digital aberrations; (2) digitalsensor improvements and nano-grids; (3) digital system improvements thatlink multiple digital corrections; (4) dynamic digital imaging systemimprovements that apply to zoom lenses and video imaging; and (5)digital image network improvements. There are several distinctimprovements for each category of sub-system listed.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic drawing of a digital imaging system.

FIG. 2 is a flow chart of an analog to digital signal process.

FIG. 3 is an image of the vignetting corrective filter in an opticallens.

FIG. 4 is a list of wide angle 35 mm lens types.

FIG. 5 is a flow chart of the digital corrective process.

FIG. 6 is a schematic diagram of different types of optical aberrationsin different configurations of the same lens.

FIG. 7 is a flow chart of the process of digital image capture,processing and storage.

FIG. 8 is a flow chart illustrating the process of selecting filtrationsettings.

FIG. 9 is a pair of diagrams showing the varied depth of fieldoptimization at different apertures.

FIG. 10 is a flow chart showing exposure optimization using digitalcorrection in camera.

FIG. 11 is a flow chart illustrating the use of in-camera specialeffects filtration.

FIG. 12 is a flow chart showing the selective in-camera filtration ofspecific objects.

FIG. 13 is a flow chart describing the exposure adjustment of an imageusing in-camera filtration.

FIG. 14 is a flow chart showing the process of in-camera special effectsfiltration.

FIG. 15 is a flow chart showing the process of in-camera digitalcorrection for sensor dust.

FIG. 16 is a diagram showing a digital camera with an ASIC array.

FIG. 17 is a diagram illustrating a digital camera with interchangeableCCD and CMOS sensors.

FIG. 18 is a diagram of a digital camera with a nano-grid before thesensor.

FIG. 19 is a diagram of a nano-grid.

FIG. 20 is a diagram showing the feedback loop connecting the processorand the nano-grid.

FIG. 21 is a diagram illustrating the tri-pixel architecture in adigital sensor.

FIG. 22 is a flow chart showing the pre-programmed modules for in-camerafunctionality.

FIG. 23 is a diagram showing the pre-sensor modification process in adigital camera.

FIG. 24 is a flow chart showing the integration of optical and digitalcorrections with feedback.

FIG. 25 is a flow chart showing the interactive feedback mechanism ofintegrated corrections.

FIG. 26 is a flow chart describing the adaptive user pattern learningprocess to process images.

FIG. 27 is a flow chart describing the feedback process in filteringimages.

FIG. 28 is a flow chart showing software privacy function in a digitalimaging system.

FIG. 29 is a flow chart showing the dynamics of zoom lens corrections indigital imaging.

FIG. 30 is a diagram illustrating object tracking in dynamic changes invideo corrections.

FIG. 31 is a flow chart showing object tracking of a digital imagingsystem.

FIG. 32 is a flow chart showing the caching process of an object inmotion in a stationary scene.

FIG. 33 is a diagram showing network coordination of fixed sensor gridtracking multiple objects.

FIG. 34 is a diagram showing the wireless communication features of thedigital imaging system.

FIG. 35 is a flow chart showing an image organization system.

DETAILED DESCRIPTION OF THE INVENTION

(I) Digital Filters to Correct Optical and Digital Aberrations

(1) Lens-Specific Digital Correction of Fixed Focal-Length Lens forSpecific Optical Aberrations in Camera.

Each lens has some sort of aberrations because of the trade-offsinvolved in producing lenses of usable size and practical commercialcost. The challenge of building lenses for SLR camera systems lies inaccounting for particular restrictions and compromises, particularly forwide-angle and zoom lenses. In the case of wide-angle fixed focal lengthlenses, several main kinds of aberrations occur, including sphericalaberration, distortion (pin cushion distortion and barrel distortion),astigmatism, curvature of field (manifesting as the reduced cornerexposure of vignetting) and comatic aberration (a distortion evidentwith long exposures). The extremes of each of these aberrations have tobe corrected in post-production.

In the case of vignetting, a type of aberration in which the corners ofan image are exposed a stop less than the image's center area, softwarecan digitally emulate a center-neutral density filter to compensate forthe light fall-off on the edges. This operation identifies the degree oflight fall-off for each specific lens and adjusts the brighter areas inthe center of the lens by appropriate exposure compensation. Aconsequence of this digital equivalent of the traditional opticalsolution to vignetting is that the image will require exposure meteringof the subject at a level consistent with the outside edges of the imagerather than the typical inner section. In the present system, the camerawill use pre-set calculations compensating (generally one stop) for thespecific gradations of the curvature of the field for each lens, with anincreasing compensation correlated to an increased angle of view.Similarly, barrel distortion and pin-cushion distortion, which alsomanifest in image corners, are corrected using processes of employingpre-set calculations, to compensate for optical distortion, particularlyin wide-angle and zoom lenses.

In addition to integrating into the camera the traditional process ofimage correction for exposure gradations of vignetting, the presentinvention offers two further embodiments. First, instead of darkeningthe center to match the darker corners, the in-camera digitalcompensation system will lighten the corners to match the exposure ofthe center of the image. In the second embodiment, the in-camera digitalcompensation system will both lighten the corners somewhat and darkenthe center of the image somewhat, to produce a more pleasing andrealistic effect. The in-camera digital corrections of the vignettingproblem dramatically improve the traditional optical solution of acenter-weighted neutral density filter, which typically degrades imagequality as it evenly illuminates the full image.

Since each lens has specific aberrations, depending on manufacturer andeven differences in the specimens themselves, the camera software systemwill have preprogrammed general corrections for each specific lens type.For instance, while a 20 mm f2.8 lens varies among manufacturers, thegeneral optical aberrations will be similar. An effective softwaresolution is one which brings the optical image quality to a levelconsistent with a benchmark as measured by low MTF curves for each lensin its class in terms of both resolution and contrast throughout theimage. To exemplify this benchmark, retrofocus rangefinder lensperformance results of wide-angle lenses, which lack a shutter mechanismto design around, can be used for comparison. To accommodate the mirrorin the SLR design type, the rear nodal point in SLR lenses are shiftedforward, creating distortion. In contrast, the present system suggestsapplying a digital solution to compensate for this problem.

In a general sense, this process of correction is similar to correctingan ocular astigmatism with reading glasses. However, rather than usingan optical solution to an ocular problem, the present system reveals adigital solution to an optical problem.

In the process, the camera identifies a specific lens and refers to adatabase that matches the lens type with the specific aberrations. Theaberrations are consistent throughout all images for each formula of aspecific prime lens type and are thus corrected by adjusting for eachspecific aberration category. Generally, the larger the angle of view ofa lens, the greater the distortion and the greater the need forin-camera digital corrections.

In addition to the kinds of distortions created in wide-angle lenses,other types of distortion occur primarily in large aperture telephotolenses, most notably chromatic aberrations that require apochromaticcorrections. Rather than employing large, heavy and expensive extradispersion glass, such as fluorite elements, the present system allowseach lens type to be digitally corrected for these types of aberrations.The in-camera digital process works by identifying a specific lens andcomparing the lens pattern to an internal database. Mathematicalcalculations compensate for the shift in red and green light thatapochromatic corrections require for very low MTF curves registeringhigh standards of resolution and contrast by emulating the opticalbenefits of extra low dispersion glass elements.

Since lenses of the same focal length but with different maximumapertures represent completely different lens designs, modifications oftheir aberrations will vary relative to each specific lens type. Forexample, a 24 mm f/2.8 lens will have a different optical formula than a24 mm f1.4 lens in a 35 mm camera system. Similarly, a 28 mm f/2.8 willdiffer from both a 28 mm f/2 lens and a 28 mm f1.4 lens and will thuseach require different adjustments for vignetting, spherical aberration,pin cushion distortion, barrel distortion and coma. In other words, eachlens with a unique optical formula will have specific aberrations, thecorrections for which will be accessible in a database.

Another type of optical aberration that affects lenses involves flare,which is a sort of specific reflection of light sources. Whileimprovements in lens coatings have been used to correct for flare, highrefractive glass also eliminates flare. The present system uses digitalprocesses to emulate these flare reduction functions on lenses even atmaximum apertures.

In addition to the lens-specific types of corrections that are suppliedby in-camera software, a function that optimizes contrast provided bylimited reflected light is required. In general, lens hoods reducereflected light. However, in the absence of a lens hood, scattered lightwill adversely affect contrast in all lenses. Thus, a general digitalsolution will optimize contrast from reflected light by emulating theeffects of a lens hood.

Because the pixels on a digital sensor behave as neutral intermediariesto record light, the aberrations on specific fixed focal length lenseswill be prominent. It is therefore necessary to filter out variousoptical impurities. In all cases, the digital in-camera software programemulates specific filters to effectively collect specific opticalaberrations.

By digitally adjusting for optical distortions, the present systemadvances the state of the art for fixed focal length lens optics, beyondany opportunities available in film cameras. The following chartillustrates a list of optical distortions that are corrected byin-camera digital solutions.

Fixed Focal Length Lens Type Zoom Lens Type Specific Wide- Wide-Wide-to- Aberrations angle Telephoto angle Tele Telephoto Spherical X XX Comatic X X X X X Astigmatism X X Distortion X X (Pin Cushion andBarrel Distortion Curvature of X X Field Chromatic X X X Flare X X X X XScattered X X X X X light (Unpolarized) Color X X X X X Accuracy No LensX X X X X Coatings(2) Multivariate Digital Correction Using Matrix Filter System in Camera

Since typically several distinct aberrations exist in a lens, it isnecessary to correct each of the aberrations. For this multi-dimensionalproblem there is a multivariate digital in-camera software correctionsolution. The problem of correcting multiple aberrations presents theadditional challenge of requiring acceleration to complete multipletasks rapidly. In most cases, the hardware employed in a camera's chipset will include an application specific integrated circuit (ASIC) whichprocesses a particular program rapidly. It is appropriate to facilitatethe combination of corrections to multiple simultaneous aberrations withan ASIC or multiple ASICs.

There is a need to optimize both resolution and contrast across theimage area for accurate light reproduction. One way to do this is tostop down the lens to an optimum aperture of about f/8. However, thissolution sacrifices the advantages of a fast lens design and capability,namely, limited depth of field and bokeh (smooth out-of-focus area).Though resolution is typically improved by stopping down a lens, digitalsensors are generally still restricted in their latitude of contrast.Therefore, regarding both resolution and contrast, it is necessary toprovide multiple adjustments of the native image with in-camera digitalcorrections.

While it is possible to produce mathematical algorithms for automaticcorrection of optical aberrations, it is also useful to have manuallyadjustable variables. Therefore, the present system includes a functionwhereby one may omit a specific correction in an image by using a lenswith multiple aberrations in order to induce a particular creativeeffect. This factor may involve a lack of exposure compensation, a lackof correction for spherical aberration or an improperly or partiallycorrected apochromatic modification.

In one example of the use of multiple corrections, simultaneousapplication of multiple digital filters concurrently corrects multipleaberrations. In effect, this is like adding layers of different eyeglasses to repair multiple types of astigmatisms for each specificocular condition. The dynamics of correcting multiple simultaneousaberrations may be complex, but since the number and type of aberrationsare constrained to a specific lens type, a centralized database may beaccessed with specific corrections for each respective aberration. Forexample, lenses with multiple complex aberrations, such as in verywide-angle lenses, will require multiple corrections. These combinationsof corrections become complex as focal length modes change in zoomlenses and as aperture changes.

The following is a list of filter types that provide digital methods ofcorrecting image problems or creating specific effects. The list is notintended to be comprehensive or systematic.

Filter Type in Other Digital App. Most Common Uses Filter Types UV (andSky) General Use Polarizer Color-enhancing (and 17 mm-200 mm (in 35 mm)Close-up warming) Contrast 17 mm-200 mm (in 35 mm) Special effectsfilters Black and White 17 mm-200 mm (in 35 mm) (Red, Orange, Yellow,Green) Infrared 17 mm-200 mm (in 35 mm) Color Graduated 17 mm-200 mm (in35 mm) (Neutral Density) Diffusion (Soft, mist/fog, 24 mm-135 mm (in 35mm) star, streak) Combinations (Neutral 17 mm-200 mm (in 35 mm) densityand enhancing, Polarizer and UV)

In the past, these optical filtration processes were added after theproduction process via editing software such as Photoshop. However, inthe present system, these combinations of processes are performedin-camera by user-adjusted settings. In the case of artificial colorchanges to an image, digital processes emulate specific optical filtersby adding a specific color or a combination of colors. On the otherhand, in the case of diffusion filtration, the in-camera digital processcreates an emulation of optical filters. The classic example of thisdiffusion approach is the soft filter, which is used for portraiture. Inthis case, various user-adjustable settings in the camera digitallymanipulate soft filtration.

(3) Depth-of-Field Optimization Using Digital Correction in Camera

In addition to correcting optically-generated aberrations with in-cameradigital processes, the present system allows in-camera depth-of-field(DOF) optimization by affecting the aperture of the lens that is used.

DOF in an image is dependent on the aperture setting in a lens, in whicha moderate DOF range—allowing a subject to be isolated in an image—canbe manipulated, that is, extended or narrowed, by the camera's digitalprocessing capability. In the film paradigm, one obtains a specificaperture, and thus the corresponding DOF, that is preset by thephotographer. However, in the digital paradigm, by contrast, one cannarrow an image's surplus DOF range in-camera by manipulating theaperture. This process can only be done in the camera, because once thedigital file is sent to post-production editing, the aperture and DOF isalready set and incapable of being changed. The aperture is narrowed incamera by isolating the subject and shifting the field of view (forwardfrom the rear range of DOF and backward from the front range of theDOF). Distance information is used to recalculate an optimal DOF. Inanother embodiment, the camera provides feedback from an internalcomputational analysis that results in a specification of less DOF andtakes another image (or images) with a larger aperture to accomplishreduced DOF at a specific focus point.

The camera will effectively identify a type of subject and provide anoptimal aperture for this subject. For instance, for a portrait, thecamera will select a shallow DOF around the subject. In the case of adistant landscape, the camera will focus on a distance at infinity andprovide a nominal aperture to correspond to shutter speed that will fitthe available light as matched to a specific lens. A near landscapephotographed with a wide-angle lens will, on the other hand, have a nearfocus and a maximum DOF; specific subjects will be highlighted withample DOF. The camera will also have the capability to bracket exposuresin several successive images based on DOF variations.

The DOF manipulation thus depends on a combination of data setsincluding the particular lens used (wide-angle lenses have greater DOFat moderate distances than telephoto lenses), the distance informationand the unique combinations of aperture and shutter speed. DOF willnarrow with less distance, with use of a telephoto lens and a fastaperture; contrarily, DOF will expand with a further distance, with useof a wide-angle lens and a slower aperture.

In another embodiment of this process, test images are taken andanalyzed, then later images taken with new settings optimize DOF foreach image type.

(4) Exposure Optimization Using Digital Correction in Camera

One phenomenon that film currently records better than digital phototechnology is exposure latitude. Film is capable of greater exposurelatitude than either CCD or CMOS digital sensors, though each digitalsensor type has strengths and weaknesses. For the crucial detailrecorded in a scene, film provides far more depth of tonal range. Yetsome of this problem—caused by the limits of digital sensors themselvesand the way that photons are recorded by electrically charged pixels—canbe digitally corrected and optimized in-camera.

The problem derives equally from the method of measuring exposure aswell as the method of image capture by a digital sensor. In general,since there is less exposure latitude in digital sensors, as compared tofilm, the maximum scope is two or three stops in the image tonal range.Consequently, the camera must meter the image within constraints of thetonal range of the digital sensor, with the sacrifice of either shadowdetail or highlight detail. In an image with broad exposure range, then,the image will generally be either too light or too dark becausemetering for one area sacrifices the other tonal category.

One way to solve this problem is to manipulate the lens aperture,because increased aperture within an optimal limit generally increasesdetail. An optimal aperture of f/8 provides more detail and clarity thanat f/2 or at f/32. The in-camera processor may thus seek out more detailin the image by manipulating the aperture to the optimal range of f/5.6to f/11, depending on the type of subject and the availability of light.With more detail in the original image, it is possible to interpolatethe digital data in the image file by increasing both shadow andhighlight detail and to gain an additional stop or two of tonal range.

In another embodiment, the tonal range of an image data set is enhancedin-camera by using meta-data to sample the range of shadow and highlightdetail. The data sets are interpolated to add requested shadow detailand/or highlight detail. Obviously, some subjects require more or lessshadow or highlight (or both), which the camera can correspondinglyadjust. These tonal range corrections are user-adjustable.

In an additional embodiment of in-camera tonal range corrections,exposure data are bracketed by manipulating the aperture and shutterspeed to lower or raise the overall exposure in one-third to one-halfstop increments. This bracketing method may be correspondingly limitedto a specific image type. For instance, in a portrait, the extraneousbackground, which ought to be out of focus, is not emphasized in theexposure data, while the main subject is carefully nuanced for balancingan optimum of both highlight and shadow, or for an exclusive emphasis oneither highlight or shadow.

The overall problem of limited tonal range in digital photography stemsfrom mismatched exposure-metering mechanisms of digital sensors withsubstantially restricted tonal range capabilities. One interestingexample of this problem occurs in scenes with two or more stops ofdifference, such as a landscape with sky on top and earth on bottom. Afiltration process will operate on the key parts of such an image, asdescribed above regarding the vignetting phenomenon; the overexposed tophalf of the scene will be gradually underexposed while the lower half ofthe scene will be gradually overexposed. This scene-specific adjustmentof exposure greatly increases the tonal range of digital images and ismade possible via analysis of the scene and comparison to a database oftypical scenes categorized by the in-camera digital processor whicheffects correction using the methods described herein. In this example,the corrective process emulates the use of neutral-density opticalfilters.

(5) Special Effects Digital Filtration of Specific Objects

Though there are several main categories of special effects opticalfilters, including color enhancing, infrared and diffusion, the use ofdiffusion filters appears to elicit the most dramatic effect. Diffusionfilters are categorized as soft effect, mist/fog, black mist, golddiffusion, and star and streak, with various degrees of diffusionproducing lesser or greater distortions. In effect, rather than removingoptically-generated distortions, we are deliberately creatingphotographically desirable distortions. It is possible to reproducethese special effects by using the digital post-capture productionprocesses in the camera. In this case, the camera digitally emulates thespecial effect by applying user-adjustable filter settings.

Portraits have traditionally used some sort of soft effect filtrationapproach which is producible in the camera using the methods describedhere. After the image is captured, the camera analyzes the image'smeta-data and applies a correction by interpolating the data withspecific filter emulation. In the past, specific camera lenses, such asthe 135 mm soft effects (also called “defocus control”) lenses performedthis function optically with an included adjustable lens element. Thisdefocus control lens type will focus on the main subject and a lenselement setting of the telephoto lens to produce a soft filter effect.In addition, because this lens type uses a nine blade aperture, thebackground that is out of focus has a pleasing bokeh in which thegradations of tone are evenly smooth. Nevertheless, a sophisticateddigital camera is able to produce the same results with more informationprovided by a normal telephoto lens, using the method of emulatingspecial effects in-camera.

Another novel special effect that is a further embodiment of the systemis the ability of the in-camera digital corrective system to use complexdata sets contained and analyzed in an image to create a threedimensional (3-D) representation of the image. The camera creates a 3-Dimage by arranging the DOF in a way that optimizes the aperture by usingdistance information and autofocus data to isolate a subject. Byremoving the foreground and background of the image as a center ofsubject focus, the DOF will emphasize the subject only as 3-D. The keyto this effect is the application of specific exposure data as indicatedabove, because it is in the increased extension of the range ofhighlight and shadow that the subject in the image will attain a 3-Dquality in contrast to its out of focus foreground and background.

An additional embodiment of the present system would extend the stillphotography in-camera special effects to video with full-motion rangesof filtration actions.

Finally, it is possible to combine different user-programmable specialeffects in-camera by adding the various types of diffusion methods for aspecific image.

(6) Selective in-Camera Filtration of Specific Objects

The combination of sophisticated auto-focus technologies and in-cameraauto-exposure systems provides the opportunity to isolate a subject byfocusing on the subject and narrowing the DOF range by manipulating theaperture. In a further extension of the subject-isolating capabilitiesof these technologies, it is possible to digitally filter out specificobjects in a scene in-camera while focusing on other selected objectsthat are in a specific range of DOF. In other words, one may applyfiltration to correct aspects of a single object or only the backgroundof a scene to the exclusion of an isolated object, rather thancorrecting a whole scene. Selective filtering of specific objects in animage by in-camera digital processing affords greater creativeflexibility.

Because the camera uses distance information to isolate a specificobject by focusing on the object within a range of DOF, it is possibleto isolate a particular object for the purposes of applying a specificfiltration preference, such as manipulating the color, correcting theoptical aberration (say, if the object is in a corner of the image of awide-angle lens), providing a special effect (such as a soft effect onlyon a specific object rather than the scene as a whole) or using somecombination of these corrections. Once the camera isolates the selectedobject (using auto-focus mechanisms and distance information), the userselects programmable correction features to perform a correctivefunction only on the specific object (or only on the parts of the scenethat are exclusive of the object). In a further embodiment,contrastively, once the object is isolated, only the background may beselectively manipulated with filtration, achieving pleasing effects.This in-camera corrective feature provides a powerful tool to rapidlymanipulate an image without using post-production editing softwaretools.

These object-specific in-camera selective filtration capabilities areparticularly dramatic with fast-moving action photography in whichsplit-second timing produces the preferred complex effects. Selectivelyidentifying a particular object for intensive combinations of filtrationis a highlight of the present system.

(7) Digital Correction in-Camera of Intermittent Aberrations Caused byDust on Digital Sensor

Dust on a digital sensor is a major concern for photographers. The useof zoom lenses compounds this condition, because as the zoom lenschanges focal-length positions, air is transmitted, which results in theproliferation and diffusion of sensor dust. Unless photography isisolated to a clean room, the problem of dust on a digital sensor willremain prevalent. The present system provides a method to correct forthis phenomenon.

In the case of dust on a sensor, a specific consistent pattern emergeson each image captured by the digital sensor. Consequently, informationfrom various images is analyzed, and the pixels affected by dust areidentified. Information from the consistent fixed pixel positions thatare affected by the dust are then isolated. The specific positions withthe dust are then analyzed by comparing the immediate areas surroundingthe dust that are not affected by it. These unaffected areas areanalyzed, and the affected areas are interpolated to provide acontinuous tone. In effect, the images identify the locations with dustby using caching technology. The continuity of the location of the dustbetween multiple images provides information to the in-camera imageprocessor to detect the specific pixel locations. The camera will thenapply a corrective process to the isolated dust locations with adjoiningexposures by interpolating these distinct locations for each specificimage configuration.

In another embodiment of the present system, “hot” (too bright) or“dead” (too dark) pixels are interpolated out of the scene using themethod described above. Unlike hot or dead pixels, dust is a similar buttemporary version of the same problem of an artifact that requiresin-camera modification. In effect, a map is built to discover, isolateand interpolate bad pixels, which are a permanent problem revealing akey limit in digital sensor technology. Separate maps are constructedfor permanent pixel dysfunctions and temporary pixel aberrations (viz.,dust). In both cases, the camera works from these maps to correct theaberrations on a pixel-level.

In a further embodiment of the present system, Monte Carlo analysis isapplied to the problem of identifying the location of dust on specificpixels (or partial pixels) by randomly creating an initial map frominformation of at least two contaminated images.

In still another embodiment of the present system, the process ofmodifying pixel aberrations (either permanent or temporary) uses asequence of operation which begins by correcting the major aberrationsfirst, then repairing the minor aberrations, thereby maximizingefficiency. This is done by starting the corrective process in aspecific location of the image and moving to other positions in anefficient pattern.

(8) Sequence of Corrections for Multiple (Optical and Digital) Types ofAberrations in Camera

Since it is evident that multiple digital filtration approaches may beused for specific types of problems or aberrations or to achievespecific effects, it is clear that a combination of the techniques maybe employed simultaneously on specific images. The present inventionallows the various optical and digital corrections to be performed incamera in a sequence of actions. The user selects the variouscombinations of functions required to be performed, inspects theeffects, and chooses the most effective combination of effects. Thus theinvention offers a combinatorial optimization of the totality ofcorrective filtration approaches.

After the images have been taken, it is possible to inspect them in thecamera using the camera's image read-out. This makes it possible tocreate new files, or to adapt a RAW file, in real time, by manipulatingthe various corrections in sequence. This post-image-capture in-cameraediting process allows multiple corrections to be applied to a range ofoptical and digital aberrations by combining various specific correctivetechniques.

In some cases, the user can pre-set specific corrections. For instance,to correct for optical aberrations, a user may leave this function onpermanently. In other cases, such as selective filtration of a specificobject or optimization for DOF or exposure, there may be discriminatinguse of specific corrective functions. In the case of selective userchoice, it is possible, by using the present invention, to select apriority sequence of corrections in layers. While specific select layersmay be permanently activated, for example to automatically adjustspecific optical aberrations, additional sets of layers may be manuallyselected in order to modify the specific aspects of each image,particularly to adjust or correct digital aberrations. This process canbe performed with a single microprocessor, multiple microprocessors,multiple ASICs or a combination of microprocessors and ASICs.

An additional embodiment of the system provides multiple combinations ofcorrections and effects via multiple independent ASICs, which onlyperform specific functions, working in parallel. The various tasks aredivided into specific-function ASICs for rapid processing. The advantageof this approach is accelerated processing speed in performing multiplesimultaneous functions.

(II) Digital Sensor Improvement and Nano-Grids

(9) Interchangeable Digital Sensor System Using Both CCD and CMOS toOptimize Best Results

Because the main digital sensor types of CCD and CMOS, like film types,each have benefits and detriments, it is sometimes advantageous toprovide the utility of both sensor types in a camera system. With theexception of a video camera, which employs three CCDs, the use ofmultiple sensors has not been adopted. Two generations ago, however, theidea of using a twin reflex camera for medium format photography wasimplemented. In this case, though focus was coupled between the lenses,one lens was used to see the subject, while the other lens took thepicture. This method was used to obtain the benefits of a rangefindercamera with a single lens reflex camera.

The use of two types of sensors in a camera is compelling, because theuser benefits from the strengths of both. In the present invention, onesensor is selected from among at least two different types of sensorsthat are rotated to an active position by the user. One advantage ofthis approach is that if one sensor experiences a problem, there is areserve sensor available at the push of a button.

This capability usefully exploits the strengths of each particularsensor. For instance, in situations in which high resolution isrequired, a CCD may be preferable, while in cases in which increasedtonal range or low noise is preferable, a CMOS sensor may be preferable.With this approach, a customer does not need to choose between differenttypes of sensors.

The process of interchanging the two chips is performed by placing thetwo chips on either façade of a plane that “flips” over (i.e., rotates180 degrees) upon demand to obtain the requirements of the chosen chiptype. This mechanism would fit behind an SLR's mirror and could easilybe performed as long as the mirror is in the “up” position. In anotherembodiment, the chip exchange process can occur by sliding alternatingchips into a sleeve from a single location and replacing thenon-utilized chip(s) into the reserve compartment. In either event, thecamera will detect the chip exchange and will automatically reprogramsoftware functions and settings for the usable chip.

(10) Nano-Grids for Selected Pixels on CCD or CMOS Integrated Circuitsto Optimize Selective Modifications of Exposure, ISO and Aberrations inDigital Photography

Digital sensors consist of arrays of pixels, arranged in rows, whichbehave as tiny buckets for converting photons to electrons. As thepixels fill up with light, they are able to discern slight differencesin color and exposure and transfer the energy, in the form of electrons,to storage. Charge coupled devices (CCDs) have been the predominant formof digital sensor because they use a form of electronic charge whichcreates the behavior of a bucket brigade of transferring data, once thebuckets in a row are filled up, to successive rows for digital datastorage of the electronic charge sets. CMOS digital sensors may bestructured with larger bucket pixels, which can increase the depth ofthe light captured and thus the latitude of light exposure that isstored. However, for the relatively larger buckets to provide increasedphoton capture capacity, it is necessary to control the width of theopening in the top and the width of the buckets so that the amount oflight captured may be modulated.

The present invention introduces a key advance in the ability of digitalsensors, particularly CMOS sensors, to modulate the size of the openingsof the pixels. Specifically, the present system provides for anano-grid, or a very small matrix of filaments, which fits over thesensor. The nano-grid is carefully calibrated to match the rows ofpixels on the sensor so as to limit the amount of light that each of thebuckets may receive. Use of the nano-grid allows a selective closing ofthe large buckets in order for photons to be restricted. Selectivemodification of specific pixels on the neutral grid makes it possible toidentify specific sets of pixels to correct for various exposure or lensaberrations.

In this embodiment of the present system, data about a specific lens areprovided to the camera in order to correct specific lens aberrations,while exposure data is used to modify image capture using nano-grids foroptimum image performance.

Nano-grids may be selectively switched at different pixel sites, akin tocontinuously programmable field programmable gate array (CP-FPGA)semiconductors, which modify architecture in order to optimize effectiveoperation by constantly manipulating the chip's gates.

Nano-grids may be used for specific image modes, for example, nocturnalimaging, which requires more time to read a sufficient amount of light.In this case, a specific software module may provide lens and exposuredata to the camera, which then determine the precise composition ofnano-grid correction to provide to specific sets of pixels on thedigital sensor. In effect, nano-filaments move to positions toeffectively block out the full capacity of the pixel buckets and thuschange the pixel effects. With use of preset nano-grid positions forparticular applications, the identification of specific vectors ofnano-filaments is performed, and exposure adjustments are made onspecific images in hardware.

The nano-grid is overlaid over the surface of the pixel architecture.The nano-grid is used not only in specific pre-set positions, but italso provides feedback to the post-capture system for analysis andrepositioning to achieve the desired effects. One effect of thenano-grid is to manually expand or narrow the range of a set of pixelbuckets; this process in turn effectively modifies not only the exposurerange but also sharpness at high ISO, thereby dramatically reducingnoise. Consequently, it becomes possible, by modifying the pixel bucketwidth and height, to obtain extremely sharp images with excellentcontrast and tonal range even in poor lighting, a feat heretoforeimpossible.

The nano-grid performs these mechanical functions by moving thenano-filaments in an arc, like expandable windshield wipers. Thoughnano-grids are particularly useful in CMOS chips, they are also usefulwith CCDs. In fact, with the advent of nano-grids, CCD pixel size (anddensity in pixel-rows which will affect the overall sensor size) may beexpanded and thus made substantially more versatile.

In a further embodiment of the present invention, multiple screens, orgrids, would be placed over the digital sensor. The use of multiplenano-grids provides increased capacity to perform the function ofclosing off the pixel buckets and, in fact, to completely close offselected pixels to make the image effect completely dark. Thecombinations of nano-grids behave as multiple screens that move left andright to achieve the desired effect. Although there is a need toperiodically calibrate the screens to effect their precise positions,this system will employ an electric charge to push the nano-filaments tothe desired locations.

Nano-filaments move to block the space allowing photons to hit the pixelin order to limit the amount of light capacity available to the pixel.The complete darkening of the pixel will result in a total black colorin the resulting image.

Exposure data feedback is provided to the digital sensor to effect theprecise positioning of the nano-grid(s). In a further aspect of thepresent system, the camera's computer will anticipate the exposure databy statistically extrapolating from the pattern created by at leastthree data sets. A microprocessor (or ASIC) controlled nano-gridmechanism will use the feedback to anticipate specific nano-gridpositions in order to optimize the exposure and corrective functions.

In one application of the nano-grid, the problem of vignetting inwide-angle lenses may be solved by activating nano-filaments innano-grid(s) primarily in the corners to correct for the darkening fromthe limits of the optical aberrations, while still maintaining very lownoise in a high ISO (low light) photographic situation. The use of thenano-grid would thus contribute to solving multiple problems.

Nano-grids will also be useful in accurately correcting for both colorand exposure detail. In fact, with nano-grids, the capacity of digitalsensors' range should be substantially increased, because the chips'pixel bucket sizes can be modulated. Therefore, not only will thelighting and color be accurate, but sharpness and optical aberrationswill also be optimized, in ways not possible before.

(11) Integrated Nano-Grids in Digital Sensor

In a further embodiment of the system, nano-grids may be integrated intothe digital sensor. In this form of the nano-grid, the nano-filamentsare constructed within the pixel buckets in order to increase theiraccuracy and responsiveness. The nano-filaments mechanically move invarious directions to perform the main operation of modulating lightinto their respective pixels. This method of organizing nano-grids andnano-filaments increases the rapidity of response to feedback. Ineffect, each pixel has a mask, or flexible lid, contained in it, whichmay open and close, to allow more or less light into the pixel bucket.

The integrated-filaments are activated by oscillation between positiveand negative charges. In the context of a CMOS sensor, the transistornetworks oscillate between positive and negative charges. Thisarchitecture allows a “push-pull” design of nano-filaments in which thenegative charge “pulls” and the positive charge “pushes” the activationof the nano-filaments. This charge-enabled nano-grid (CENG)advantageously allows modulating gates (i.e., filaments) integrated intothe pixel to reduce spaces between pixels, thereby allowing more pixelsto be packed on the same surface area. The net benefit of the use ofintegrated CENG filaments is that specific sets of nano-filaments willproduce specific effects on-demand and allow far more tonal detail thanhas been possible before.

In a further embodiment of the present system, sophisticated digitalsensors may contain combinations of nano-grids that appear on top of thesensor as well as nano-grids that are integrated into the digitalsensor. This combination will provide maximum latitude for processingthe greatest effect available.

(12) Combinations of Nano-Grids and Digital Corrections Applied toDigital Imaging System

Whereas it is possible to exclusively implement nano-grids to controlthe amount of light penetrating specific pixels, and it is possible toexclusively provide digital corrections as specified above regardingcorrecting optical or digital aberrations, a further embodiment of thepresent invention combines the two processes in order to optimizeimaging.

Combining these two complex processes makes it possible to modify pixelcapacity to maximize exposure latitude, to expand exposure modificationand to apply digital correctives for optical and digital aberrations.Hence selective exposure far beyond the limits of present film ordigital photography is made possible. The restrictions of film can thusbe transcended by using the present system, whereas use of a static andlimited digital system would not be sufficient to facilitate thesecomplex corrections.

The unique combinations of these processes also illustrate a complexsystem that provides feedback from both the environment and thephotographer. The photographer may select preset exposure settings thatwill activate a range of options in both the nano-grids and the digitalcorrective system, while the lens aberration corrective system isautomatically implemented. Once the camera detects specific conditions,such as a broad range of exposure latitude, from very bright to verydark, in the scene, it computes the precise number and location ofnano-grids needed to modulate the pixels for optimum exposure withhighlight and shadow detail and extreme sharpness, even in relativelylow light. The dynamics of these multiple processes present trade-offsin selecting the best available set of selected modifications.

(13) Tri-Well Pixels

As indicated above, one of the key problems with current digital sensorsinvolves dynamic range. There is a need to limit the scope of the spacein the pixel well, into which light is captured, then converted intoelectrons. The challenge with current technologies is to balance detailsin shadow and highlight areas, particularly to achieve low noise atrelatively high ISO speeds.

In addition to the concept of nano-grids, both in surface screen andintegrated embodiments, as specified in (10) to (11) above, the presentsystem introduces the notion of three side-by-side differentially-sizedbuckets within each pixel intended to optimize dynamic range forincreased sensitivity. In the most common configuration, the threedifferent-sized buckets are arranged with the largest bucket (in bothwidth and height) in the center, with the second and third largestbuckets on either side. The buckets are elliptical and concave inarchitecture to increase efficiency of fitting together in a round pixelstructure. Their structures are semi-circular and elongated. The largestand tallest bucket will be tasked with maintaining the details inhighlights, and the smallest will be tasked with maximizing the detailsin shadows, while the mid-sized bucket will be tasked with maintainingthe middle range of exposure details. The pixel will have data from allthree buckets available, but will select the data from one or morebuckets depending on the exposure details.

The system is analogous to the high fidelity sound technology inspeakers with crossovers, whose several frequencies are used by thetweeters, mid-range(s) and woofers; the crossover point at which thefrequency changes from one component to another can be modified based onthe specific mechanics of each component.

In the case of the multiple buckets in a single pixel, the buckets areconnected by filaments to a central grid which captures and stores theelectrons. When the photographic scene displays increased light, imagedata from the larger buckets are selected to be recorded by theprocessor, while in cases of darkness and increased need forsensitivity, the smaller buckets are selected to be recorded; themid-sized bucket is used in normal light situations of most cases.Further, this multi-aspect architecture can use pixels in varyingpositions on the sensor differently, particularly to facilitateprocessing far more dynamic range and to produce uniform tonal range inscenes that vary more than two or three stops. This novel multi-aspectmodel solves a number of key problems involving exposure dynamics indigital photography.

In another embodiment of the system, there may be more than threebuckets in a pixel, so as to divide out the functions further and createeven finer tonal continuity. In a further embodiment of the system,several pixels in a super-pixel allow red, green and blue colors to besegregated by each sub-pixel. This approach will be useful particularlyin CCD semiconductors because of limits of this architecture, whichrequire coupling circuitry between pixels to pass a charge between rowsof pixels. In this case, outputs will vary between the micro-pixels tofacilitate the differential processing required.

While cases of two side-by-side pixels might solve these exposurelatitude problems, they represent an inadequate solution, much as aspeaker with only two components limits the dynamic range outputdramatically in contrast with a speaker with five components. This issimilar to comparing a diode and a transistor.

(III) Digital System Improvements that Link Multiple Digital Corrections

(14) Auto Pre-Programmed Modules for Specific Functions in DigitalImaging System

To process the functions specified in this integrated digital imagingsystem, it is necessary for automated pre-programmed modules to detectthe specific lens type and the specific digital sensor(s) used to assessthe appropriate corrections or alterations. The purpose of thepre-programmed modules is to access a preset library of (a) typicalcorrections of lenses, (b) typical scene types with appropriate exposuremodes, (c) specific effects that may be selected and (d) specific sensorfunctions. It is important to match a particular lens to a particularsensor type so that adjustments are calibrated to this pairing. Theprocessing software is stored in either a microprocessor or an ASIC inorder to process the images after they are captured by the sensor butbefore they are transferred to storage on a memory device.

In another embodiment, the system processes image corrections after thedigital data is stored and constantly accesses the original stored datafile in the production of a corrected file. This process allows forimmediate data processing and storage which affords more time toaccomplish specific corrective functions. There are thus cases whenreal-time correctives are neither necessary nor possible. Such increasedprocessing capability may also facilitate a more complete correctivetask.

In an additional embodiment, because similar correctives and effects maybe provided to images that share the specific combination of lens andsensor, in order to accelerate the process of optimizing the images,batches of similar images may be processed together. This batchprocessing method may include the creation of duplicate images for eachimage captured, including a RAW image that contains no changes to thenative image capture and a simultaneous auto-corrected image. Theoptimized image may be simultaneously compressed, to maximize storagecapabilities, while the RAW image may be left uncompressed so as tomaintain original detail.

(15) Apparatus and Process for Affecting Pre-Sensor Optical and DigitalCorrections in Digital Imaging System

Given the nature of light transmission, not all optical corrections areoptimized by modification after the image is captured by the sensor.Though a range of important corrections and effects may be made afterimage capture, such as correction for optical or digital aberrations,there are several types of corrections that are required to be madebefore the light reaches the sensor. One example of this pre-sensordigital correction involves the use of a low-pass or anti-aliasingfilter that resides in front of the digital sensor to minimize moiré andaliasing digital problems (although the use of this filter adverselyaffects image sharpness).

In the case of optical corrections, one class of filter that requiresuse before the digital sensor is the polarizing filter, because oncelight is captured on the digital sensor, the polarizing effect will notbe available. Another type of correction that involves use of a filteror lens before the digital sensor is the close-up filter. This lattersolution allows a lens's closest focusing plane to be closer to thefront of a lens and has the effect of diminishing the rear plane of thedepth of field. The close-up filter may be optimized for use withfloating rear-element group lenses which allow increasingly closefocusing. In one embodiment of the system, specific pre-sensor opticalfilters may be used to provide polarization and close up corrections.The use of in-camera optical (circular) polarization would helpstandardize this valuable process and eliminate the need to maintainseveral external polarizer filters for each lens mount.

Since the present system entails an embodiment which uses nano-grids toperform specific exposure modifications before the light hits thedigital sensor, it is possible to use these nano-grids for theapplications of polarization and close-up filter. These filtrationcapabilities occur between the lens and the digital sensor.

In order to optimize the use of pre-sensor filtration, an image isinitially tested and analyzed before the optimized corrections areactivated and the pre-sensor changes are made. This process is analogousto the use of automated flash photography in which a feedback mechanismis provided; a scene is evaluated, and the initial flash data analyzedand modified to correspond to the correct exposure before a final flashis produced.

Because the camera system processes post-capture data, in order tooptimize images for optical and digital problems, as well ascontinuously makes changes to pre-sensor filtration, multiple ASICs workin parallel to make the conversion of the image after capture. The useof parallel ASICs to perform specific correction processes solves theproblem of capturing images and making post-capture corrections whilesimultaneously adapting the pre-sensor filtration system.

As an alternative embodiment of the system, a microprocessor (andsoftware) may perform specific pre-sensor adjustments while the ASIC(s)performs specific corrective functions. In another embodiment, theASIC(s) may perform the specific pre-sensor adjustments while amicroprocessor (and software) will perform the specific correctivefunctions.

(16) Integrated Digital Imaging System for Optical and DigitalCorrections with Feedback Dynamics

Because the present system consists of, and uses, complex sub-systems,including an auto-focus mechanism, an auto-exposure mechanism, a shuttermechanism, an automatic flash mechanism, a digital sensor mechanism, adigital processing mechanism and a digital storage mechanism, it ispossible to realize interaction dynamics that contain feedback. Theinteractive process of operating these sub-systems involves a learningprogression. The image is analyzed, solutions are tested and an optimalsolution is selected and implemented, all in real time. By choosing aspecific logic vector in a decision tree involving an initial variable,the process begins again with another key variable in real-time untilthe final image is captured and optimized.

In order to accomplish these complex processes, specific variables, suchas aperture data, shutter speed data, lens data, digital sensor data andsubject type are registered and analyzed by the camera. As environmentaldata changes, the camera mechanisms adapt to the environmental and thephotographer's situation.

In order to accelerate these processes, the camera learns to anticipatethe user's behaviors, the user's preferences and the subject'sbehaviors. By providing user-adjusted setting modifications for opticaland digital corrections, the camera establishes a reference point forprocessing rapid image changes. In particular, the camera's softwarewill analyze trends in the user's pattern of behaviors and preferencesas well as pattern changes in the subject's behaviors. Anticipationprocesses are programmed into the autofocus and automated flash systemsbecause of the extremely rapid reaction-time requirements of thesespecific mechanisms.

In one embodiment of the system, a method of processing a chain of rapidimage captures is to employ computer-caching techniques in which a firstimage is processed in a normal way while later images are processed inan accelerated way. This is possible because the first image providesdata to the system to analyze; these data then allow the system toanticipate further similar images and to use similar auto-focus andauto-exposure data. The optical and digital corrections are performed ina batch fashion by applying similar changes to near-identical imageproblems in order to dramatically accelerate the processing speed of achain of images. This caching and anticipation approach is very usefulin fast-paced action photography.

Another embodiment of the process of rapidly capturing a chain of imagesin sequence employs multi-threading techniques. Dividing the functionsbetween specific subsystem ASICs allows multiple corrections to beperformed in a parallel cascade for efficient task completion. Oneadvantage of breaking down functions to specific processors is theacquired ability to start on one function and, while the system is inthe process of completing a task, to begin other tasks. This processeliminates the lag between the specific subsystems.

(17) Adaptive User Pattern Learning with User-Programmable Functions inDigital Imaging System

In order to optimize its functions, the camera needs to learn about theuser's preferences. When the user uses the camera, the camera evaluatesthe use patterns. Since the camera is programmed with a database ofcommon user patterns, it can identify common uses and anticipate commonuses of similar users by employing a “collaborative filtering” mechanism(i.e., if you like this camera setting, you should like this othersetting because similar users who have liked the first setting have alsoliked the second setting). By anticipating common uses of each camerauser, the camera optimizes its functions for each use and for each user.In effect, the camera's learning of user preferences is a sort of guidedprocess of experimentation. Evolving algorithms learn about the userfrom actual use patterns.

One positive effect of this learning process of the camera about theuser's patterns of behavior is that the filtration process becomesadaptive. The camera builds an initial map of the user's preferencesfrom the user's actual selections. From the starting point of commontypes of personal selections, the camera uses standard templates of maintypes of uses that are fulfilled for each user's applications. Forinstance, if a photographer typically takes portraits with a traditionalportrait lens, the camera will be aware of this and will activatefiltration processes that are optimal for this type of portraiturephotography, such as instilling limited depth of field on a subject andout-of-focus foreground and background. Contrarily, if landscape imagesare selected, depth of field will be increased substantially and thelens focused on either infinity or a medium point depending on thespecific type of subject matter. The camera builds a model for each userbased on the accumulation of experience.

In order for the camera to learn about the preferences of a specificuser, the camera must adjust to each particular user, much as eachindividual identity must log onto a computer network privately.

Since the dynamics of the combined subsystems are complex, and adaptive,it is necessary that automated adjustments be interactive. Oncedetection of the lens type, the sensor type, the exposure settings, theuser and the subject is made, optical and digital distortions areidentified and specific combinations of corrections are applied bothbefore and after the digital sensor in order to optimize the image. Allof this is accomplished in less time than the blink of an eye.

(18) Software Privacy Function in Digital Imaging System

Because digital camera systems are able to use software and wirelessmechanisms for their operation, it is possible to activate aspects ofthe camera remotely. Conversely, it is possible to disable operations ofthe camera remotely.

The present invention embodies a capability to externally disable thecamera remotely in specific locations that require privacy, such assecret government areas (courthouses), private homes or businesses thatare image-free zones. In these cases, a signal from an external sourceis provided to disable the shutter from firing. This black-outcapability will allow external control of access to specific sites. As acondition of access, only a camera with this feature may be admitted topublic buildings, so that even if the camera is permitted to operate,permission is only conditional. For instance, the owner of the buildingmay allow the camera to function only in a specific set of rooms but notin others. Cameras without this feature may not be allowed in privatespaces where control must be externally restricted.

This blocking feature will require the addition of specific blockingsoftware, which may be automatically downloaded as one enters specificbuildings. Similarly, in order to be granted permission to access thecamera, or specific functions of the camera, the downloading of a “key”may be required.

Moreover, a further embodiment of the system may make it necessary todownload software keys to get access to filtration capabilities in thecamera in order to obtain optimum images. For example, the user may berequired to pay a fee to download software in real time that will permither to access a particular function in the camera to obtain a criticalimage. A spectrum of quality and complexity in filtration capabilitiesmay be made obtainable for a range of fees on-demand. Therefore, theexternal downloading of software for the camera need not be limited to ablack out function.

(IV) Dynamic Digital Imaging System Improvements that Apply to ZoomLenses and Video Imaging

(19) Dynamics of Zoom Lens Corrections in Digital Imaging System

Whereas the optical aberrations of prime (fixed focal length) lenseswere discussed above, the modulation of optical aberrations of zoomlenses is another problem to consider. As a wholly different species oflens, zoom lenses have become extremely complex optical mechanismsconsisting of multiple groups of lens elements. The general problem withzoom lenses is the trade-off that must be made: To minimize thedistortions of the widest possible focal length, distortions becomemaximized at the longest possible focal length, and vice-versa.Consequently, zoom lens architecture is inherently compromised on imagequality. Over the years, lens designers have developed lens formulasthat have dramatically improved image quality and that compete withtypical prime lenses. As an example of this evolution in quality, theclass of 70-200 f/2.8 35 mm zoom lenses, now in their sixth generation,has supplied substantial improvements over earlier telephoto zooms.However, in general, zoom lenses have more aberrations than primes andthus require increased optical corrections. The need to solve theproblem of zoom Jens aberration correction is accentuated by theirincreased use in photography because of their simplicity andversatility.

The dynamics of the zooming process present specific difficulties forthe purposes of correcting optical aberrations in digital imagingsystems. With fixed-focal length lenses, the camera can detect the lensand provide an immediate consistent modification for a varying range ofapertures. In the case of zooms, however, where the focal-length is notfixed, the adjustments must correlate to the changes in the focallength. In effect, this condition presents a continuous resamplingprocess. When combined with changing scenes, the zooming processrequires far faster responses to changing inputs by the camera system.This process resembles the tracking of a kaleidoscope's changing imagestructures as the wheel on the device is constantly turned.

In order to solve the problem of distortion at the wide-angle part ofselect zoom lenses, manufacturers have been using aspherical elementswhich are complex shapes that require special production techniques. Onthe other hand, in order to solve the problem of chromatic aberration inselect telephoto lenses, manufacturers have used extra low dispersionglass elements, particularly at the front of the zoom lens. Since thereare generally three main classes of zoom lenses—wide-angle towide-angle, wide-angle to telephoto and telephoto totelephoto—aspherical elements have been used in wide-angle zoom lenses,while extra low dispersion glass has been used in the telephoto zoomlenses and both kinds of lens elements have been included in thewide-angle to telephoto zoom lenses.

The changing focal lengths of zoom lenses add a variable to the complexset of variables of the interacting sub-systems in the digital imagingsystem. The digital camera system must therefore track the movement ofthe changes in the focal lengths in zoom lenses and continuously makemodifications to the varied optical aberrations in these types oflenses. Unlike in fixed focal length lenses, the aberrations change atdifferent focal lengths in zoom lenses, and the camera must track thesechanges.

The present system is designed to make the corrections to these changingaberrations in zoom lenses by noting the changed focal length atspecific times of each lens. For a fixed focal length lens, the camerarefers to a database of information to provide information to correctspecific types of aberrations; for a zoom lens, the camera's databasecontains multiplex information for each focal length in each respectivezoom lens type. This is as if each zoom lens contains a combination ofmultiple lenses of specific focal lengths. When the zoom is moved to anew focal position, the camera reads the lens as a specific focal lengthand makes corrections to aberrations based on this specific setting.Although the camera reads the zoom lens at a specific moment in time andadjusts the necessary modifications to correct for aberrations at thatspecific focal length at that time, overall the zoom lens requires thecamera to rapidly make these adjustments.

Since zoom lenses employ dynamic processes of change, it is possible totrack a moving subject in real-time by changing focal length from astationary vantage. These changed focal length positions are tracked bythe auto-focus system, but also by the auto-exposure system in thecamera. The present system thus allows for zoom tracking in order toanticipate the direction of zoom lens changes, much as the focus on themoving subject involves focus tracking mechanisms. These systems usefuzzy logic and evolutionary algorithms to anticipate the movement ofthe subject and thus of the focal length change of zoom lens. In thisway it is possible to accelerate the lens aberration correction processusing zoom lenses.

Because the zoom lenses typically increase aberrations precisely becauseof the lens design compromises, these types of lenses are ideally suitedto the present digital imaging system. The present system allows thezoom lens to be used at high quality without needing to stop down theaperture, thereby resulting in superior photographic opportunities.

(20) Dynamic Changes in Video Corrections of Digital Imaging System

While the zoom lens presents the need to provide a dynamic solution tothe process of making corrections to optical aberrations, videophotography provides another case of a process that requires dynamicsolutions. The same principles that apply to still photography apply tovideo; auto-focus variability, aperture and depth-of-field variabilityaspects, shutter speed variability aspects, differences in lens focallength and artificial lighting variability suggest that video be viewedas merely a very rapid (30 to 60 frames per second) application of stillphotography. Nonetheless, video presents new classes of dynamicproblems, most notably regarding the matter of tracking changingsubjects in real time.

The process of shifting subject positions, even if the camera isstationary, presents a change of multiple variables that require theautomated subsystems (auto-focus, auto-exposure, auto-flash, etc.) to beintegrated. Feedback is presented by subjects in the externalenvironment with changing focus and exposure variables. In these cases,even with a modulating shutter speed, the three main variables of changeare a zoom lens to continuously change the focal length, auto-focus totrack a subject and aperture modifications to continuously changedepth-of-field.

The unique dynamics of these complex sub-systems presents particularchallenges for a digital imaging system to produce rapid results withthe use of advanced ASICs and microprocessors. By incorporatingtechniques that track objects with advanced auto-focus mechanisms,anticipate zoom lens changes and predict optimal exposures as well asmake automatic corrections to both optical and digital aberrations inreal time, the present system continuously optimizes the video imagingprocess.

(21) “Stationary-Scene Object-Motion” Caching Process, with Applicationto Video, in Digital Imaging System

Because video imaging processes employ full motion activity,particularly of subjects in the environment, tracking a subject in avideo system is problematic. Once a subject is identified and selected,the subject is automatically tracked with auto-focus and auto-exposuremechanisms by a zoom lens apparatus. There is a particular need toidentify and track a subject within a broad stationary scene.

The present system accomplishes this task by using anticipatoryobject-trajectory tracking. The parts of the stationary scene that arenot being tracked are cached. In other words, precisely because thebackground of the scene is stationary, this part of the scene is nottracked for focus or exposure. On the other hand, the object in motionis identified and tracked by subtracting the extraneous data of thestationary scene. Multiple objects are tracked by comparing data aboutthese combinations of objects and their relations and determining theappropriate exposure and focus settings.

While Monte Carlo processes use random settings to self-organize aninitial map, which are useful as a baseline for the purpose ofanticipating tracking data sets, the present system subtracts the knowninformation about the specific object(s) being tracked from thestationary background in the environment. In other words, the backgrounddata is “blanked out” in a caching process while the main subject(s) aretracked. By so using these techniques, the camera system can efficientlycalculate the modifications needed to optimize the video scene.

In a further embodiment of the system, a chip-set is enabled in videodisplay devices (i.e., video monitors) to implement select correctionsfor optical and digital distortions. The user may modify settings forautomating the process of achieving optimum video images.

(V) Digital Image Networking

(22) Network Coordination of Fixed Sensor Grid to Track Multiple Objectsin Digital Video Imaging System

While the previous discussion has focused on employing a single camerato capture images, the present system is also useful for networkingsensors in a sensor grid in order to track multiple objects.Specifically, the present system may be used in surveillance andreconnaissance situations to track objects over time. Using a grid ofimage sensors with overlapping range parameters makes it possible toorganize a complex network of sensors for surveillance activities.

After selecting specific objects to track, the system follows theobjects as they move from location to location, appropriately modulatingthe focus, the lens focal length, the ISO and the exposure settings. Asthe subject moves from one section of a grid to another, the sensors arecoordinated to “hand off” the object to other sensors on the grid, muchlike a cellular phone network hands off calls between cells.

In another embodiment of the present system, the cameras in the networkmay be mobile instead of stationary in a fixed sensor grid. In thiscase, self-organizing aspects of the mobile sensor grid track mobileobjects in real time. One application of this complex system, whichdraws on earlier work in collective robotics, is in cinematography,which requires multiple transportable perspectives of mobile subjects.The complex dynamics of a mobile sensor network provides complexfeedback in this manifestation of the present system.

(23) Automatic Wireless Off-Porting of Back-Up Images to External DataBank

Because the present system uses digital files, it is possible to movethese files to an external site for storage. The present system hascapabilities to off-port images to an external data bank automatically.This feature is valuable in order to preserve on-board storagecapability.

Whether implemented in a local area network (LAN) or a wide area network(WAN), by using a built-in wireless router, the present digital imagingsystem may be set to send data files directly and automatically tohard-drive storage either in a device in the same room or uploaded tothe Internet for storage around the world. This capability is criticalfor managing massive files of large sensor data sets and preservingvaluable in-camera storage space. When automatically sending data filesto a nearby computer, the computer may act as a data-port relay toautomatically resend the images to an Internet site for storage. Thesystem will maintain the option of keeping some images in the camera andsending duplicate copies of digital files of images to another site forstorage as a backup. This automatic back-up process provides insurancefor the photographer.

In another embodiment of the system, just as image files are off-loadedto external storage, software files are periodically downloaded to thecamera in order to update the camera settings and the database system.For example, as the camera manufacturer provides new lenses for thecamera, it becomes necessary to load new updated settings to accommodatecorrections for the new lenses. Similarly, as the camera requires newsoftware updates with improved algorithms to further optimize thecorrective functions of both the optical and digital mechanisms, thecamera will automatically accept these. This feature is particularlyimportant to both manufacturer and user because the ability to updatesoftware capability periodically will protect a user from needing toupgrade major hardware such as with a lens replacement.

(24) Image Organization System

The present digital imaging system does not merely allow for the storageof image data files on external storage. Because of the problems ofprotecting storage and the need to make multiple back-ups in the digitalsphere, it is also necessary to store digital image files in multipledatabase locations. The images are organized in a main database byvarious criteria, such as time, location, subject-type, etc., and thenrerouted to various locations around the world for safe storage. Whilespecific sets of images may be stored together, the need to identify thelocations is less important than the need to have control of the maindatabase list which identifies the locations.

In order to maintain security, the digital imaging files mayperiodically be rotated randomly between locations. Only the maindatabase list, which is constantly updated, maintains information ontheir location. In fact, specific digital bits of a single image may bemaintained at different locations in order to maintain further security.Thus, on many computers around the world bits of each image may bestored, and continuously rotated, with constantly updated registriesmaintaining their complex hybrid whereabouts. These rotation storagefunctions are performed by a randomizer logic engine.

In a further embodiment of the present system, once the digital filesare off-loaded from the camera system to external storage, specificimages may be automatically identified and further specific correctionsautomatically provided.

In yet another embodiment of the system, the images that are off-loadedfrom the camera to the external storage system are organized accordingto various criteria, such as accuracy of focus or exposure or quality ofimage type, in order to be automatically prioritized. The camera, withthe assistance of an initial setting of user priorities, willautomatically order new images with a higher or lower priority relativeto other images and camera settings. Thus, at the end of a day, theimages may be displayed in an order preferred by the user. Lesser imageswill be automatically routed to a lower position as they do not meetspecific criteria, and better images will be routed to a relativelyhigher position in the organization of files. This feature ofautomatically assisting in the organization of the digital image filesis a very useful one which will save photographers time.

(25) Wireless Digital Image System Automatically Generating Prints fromImage Capture

The present digital imaging system not only automatically off-loadsdigital image files to remote locations for storage; the system alsowill allow one to photograph an image (or sequence of images) in onelocation (i.e., Paris) and instantly print it in another location (i.e.,Los Angeles) for publication in real time. In addition, an image may becaptured by the camera and instantly uploaded to a pre-programmed Website for publication by using wireless technologies. In addition, it ispossible to automatically print the digital image file anywhere in theworld virtually the moment the image is taken. This system makes thisinstantaneity particularly possible precisely because the imagecorrections are automated in-camera. Since there is no need in mostcases to further edit the image files, they are thus generally ready forimmediate release.

General Architecture and Dynamics

FIG. 1 illustrates the overall structure of the system. Object data(100) in the domain of objects being photographed by the cameraconstantly change. In some cases, the lighting changes, while in othercases, the positions of the objects change. At 120 the flash willartificially light the objects. Flashes may be either on the camera orremote from the camera. A camera sensor (130) will detect external datachanges. An optical lens (110) feeds analogue imaging data to thedigital sensor(s) (175 and 180). In some cases, a mirror (170) willswivel in order to directly input imaging data to the digital sensorafter the shutter (150) is fired. In some cases, a filter (160) standsbetween the lens and the digital sensor(s). A digital signal processor(DSP) (195) is connected to at least one sensor (A or B). A database(190) is connected to at least one sensor (A or B) as well. The digitalsensors are connected to an ASIC (195) and/or a microprocessor (193) inorder to process the image and control the camera. The digital imagedata is passed from the digital sensor to either the ASIC ormicroprocessor and then stored in memory (197). The presence or absenceof specific elements of this mechanism is not required for the correctfunctioning of this system. Consequently, a number of the drawingscontained herein will focus only on specific sub-assemblies of theoverall digital imaging system mechanism.

The present invention is intended to operate with a spectrum of cameratypes. These camera categories include digital still cameras without amirror mechanism or without an optical interface. The present systemapplies to cameras with single lens reflex mechanisms. In addition, thepresent system applies to video cameras, both with or without mirrormechanisms, including camcorders. Finally, many of the functionsdisclosed in the present system are integrated into specific imagingsensors. The system applies to image sensors that are integrated withcomplex system functions, including those described herein, with “systemon a chip” (SoC) capabilities in a single microelectronic integratedcircuit. The invention also applies to networks of sensors, networks ofcameras or integrated networks of both sensors and cameras.

FIG. 2 describes digital image signal processing. After an analog signal(200) is captured, it is converted to a digital signal by ananalog-to-digital converter (210). The signal is then forwarded to thedigital signal processor (220) and filtered (230). The digital signal isthen stored in memory (240).

In FIG. 3, the correction to the optical vignetting problem is shown. Invignetting, the lens (300) makes the edges of the image significantlydarker as the light falls off, a phenomenon particularly prominent onwide angle lenses. As represented here, the concentric circles offiltration correction are darker in the middle (310) of the image inorder to counteract the vignetting effect. The overall image aperture isreduced from one half to two stops in order to compensate for themaximum light fall off depending on the amount of vignetting effect fromeach lens type.

FIG. 4 shows a list of several different wide angle lenses in adatabase. In the 35 mm domain, the 14 mm (400), 20 mm (410), 24 mm(420), 28 mm (430) and 35 mm (440) lenses, each with f/2.8 apertures arelisted in the database. Each lens type presents a distinct formulationinvolving different sets of optical data that require differentaberration corrections.

The digital corrective process is described in FIG. 5. After a specificlens type is identified (500) and a database is referenced (510), thelens type is matched with the specific optical aberrations (520). Thedigital filter then applies a correction to specific optical aberrations(530). As an example, the vignetting effect is corrected by graduallydarkening the center of the image in concentric rings (540).

FIG. 6 shows the database configuration of several different types of 24mm f/2.8 lenses (600 to 630) with different optical configurations. Eachlens type has a different set of manifestations of optical aberrationsincluding vignetting (640), spherical aberration (650), pin cushiondistortion (660), barrel distortion (670) and comatic aberration (680).For each lens, there will be a unique combination of optical aberrationscompared to each lens type from different optical configurations. Thedatabase is accessed to provide the multi-objective optimizationsolution for correcting several different optical aberrations for eachspecific lens formula.

In FIG. 7, after a camera captures an image (700) and uses a digitalsensor to create a digital file (710), the digital file is forwarded tothe digital signal processor semiconductor (720). The DSP applies thefiltration (730) to correct the optical aberrations from the lenses. Therevised digital file is then sent to storage (740).

Digital filtration is performed by employing the DSP hardware as well asspecific software in order to attain specific aberration corrections. Inan optical filter, which typically sits at the front of a lens andperforms a single function of modifying the optical characteristics ofthe lens, the electronic filter will process the image after it isconverted from an analogue representation to a digital signal. Commondigital filters include a low pass filter or anti-aliasing filter. Inmost cases digital imaging filtration is a discrete time application andis processed in a time-signal sequence.

One example of a digital filtration process is a fast Fourier transform(FFT). The digital signal is modified by applying an algorithm toextract the frequency spectrum. The original function is thenreconstructed by an inverse transformation of the original signal. Thesignal can be manipulated to perform various conversions. This processis used to sharpen or soften an image. For instance, by differentiatingthe frequency spectrum, the high frequency can be emphasized by limitingthe low frequency, as in a high pass filter. Digital filtration istypically performed by the DSP after the image is captured and beforethe image file is stored. However, in the present system, there is somefiltration before the digital sensor that captures the image as well assome filtration processing after the sensor sends the file to the DSP.

In order to accelerate the filtration process, the digital file will bebroken into parts, with each part processed simultaneously. Filtering aone-dimensional image will treat data from each column of a digitalsensor separately. When the data is treated like a two dimensionalimage, the data file may be treated with different techniques. Forinstance, different quadrants of the image may be analyzed and filteredseparately. In addition, the highlights and the shadows in the variedfrequency range may be analyzed and filtered separately as well.Similarly, a two dimensional image file may be analyzed by starting in acorner and working in each contiguous quadrant in a circular (clockwiseor counterclockwise) order. Further, the filtration process may begin inthe corners and work inwards or begin in the center of the image andwork outwards. For instance, in wide angle lens filtration to correctoptical aberrations, the outer edges will be the most prominentdistortions that will require the most corrections; therefore, thefiltration process will work by starting on the corners first.

The present invention also addresses the multi-functional corrections inan image by applying multiple simultaneous techniques. This is doneeither by performing a sequential filtration process or a simultaneousfiltration process. In either case, the image is re-filtered to makemore than one pass in order to correct different types of aberrations.

Different types of aberrations require different types of filtration. Inthe case of pin cushion distortion and barrel distortion, which areinverse appearing aberrations, the filtration process will adjust theedges of affected digital files captured with wide-angle lenses. Theoptimized images will be accessed by the database and compared to theactual image files. The filtration will be applied to each image file toclosely correct the distortions bit by bit. In effect, the correcteddigital images will be reverse engineered to discover the uniquedistortions as they establish a pattern by comparing the input digitalimages and the database of optimized images. The digital imagecorrection will be applied once the aberration is assessed.

Each lens provides data to the camera microprocessor and DSP about itsunique characteristics. The lens is pre-programmed with aberration datapertaining to that lens type and even to each particular lens(ascertained through a testing process). The lens then provides thisspecific data to the camera for processing of optical aberrations. Inone additional embodiment, the lens will also contain software tocorrect its aberrations that will also be sent to the camera processorsin order to be applied to specific digital file filtration. Asinformation and techniques are made available, new software to ascertainand correct each lens's unique optical aberrations will be forwarded tothe camera and stored in the lens, thereby providing an upgrade path tocontinuously improve the optical qualities of lenses by employing a sortof after-manufacture digital correction.

FIG. 8 shows the process of selecting filtration settings. After theuser selects the specific filtration settings (800), the user marks thespecific filtration setting and accepts the setting (810). The user maycontinue to select multiple settings. When the user has selected allpreferences (820) and accepts the settings the user returns to the mainmenu.

In FIG. 9 varied depth of field optimization at different apertures isdescribed. In the first drawing, the object (930) has a depth of field(940) that has a broad range of clarity around the object. The camera(900) evaluates the distance data (920) and the object to establish anoptimal depth of field range between f/8 and f/11. In the seconddrawing, the object (980) has a narrower depth of field range (990)based on distance data (970) and object data from the camera (950) todetermine an optimal depth of field range between f/2.8 and f/5.6. Thecamera's automatic alteration of the aperture to narrow the depth offield is based on a range of factors, including the focus on the object,the object's motion, the distance to the camera, shutter speedconstraints and the light on the object. Modulating apertureautomatically in the camera provides a blurring of the background andthe foreground so that the object can stand out without distraction.

Exposure optimization using digital correction is shown in FIG. 10.After the camera assesses an object's distance from the camera (1000)and identifies the optimal aperture of the image to maximize the dynamicrange (1010), the aperture is either increased (1020) or decreased(1030) and the correction applied (1040).

In-camera special effects filtration is illustrated in FIG. 11. Once theimage exposure is assessed by using the camera exposure meter (1100),the camera corrects the image by either underexposing the image by onethird stops to one stop if the image is overexposed (1110) oroverexposing the image by one third stops to one stop if the image isunderexposed (1120). The camera finally takes a picture and stores theimage (1130).

FIG. 12 shows the selective in-camera filtration of specific objects.After the camera identifies specific objects (1200), the camera assessesdistance to the objects (1210). This is done by sending out an infraredlight signal, bouncing it off the object, and measuring the distancedata from the camera to the object. The camera isolates the objects on aone-dimensional Euclidian plane (1220) and then selects the optimumaperture to isolate objects and to blur the background and foreground(1230). The camera tracks the motion of objects (1240) and activates adigital sensor (1250) thereby capturing the image. The image data issent to a post-sensor processor (1260) where the camera applies specificfiltration only to the designated objects (1265) and stores the file inmemory (1270). Alternatively, the camera will apply specific filtrationonly to the background (not to the objects) (1275) and then stores thefile in memory (1280). The effect of only applying filtration to anobject or to its background is to isolate the subject. The type anddegree of special effects, which are selected by user-adjustablein-camera software, will be determined by the user's palate offiltration choices.

Image exposure adjustment using in-camera filtration is described inFIG. 13. Once the camera assesses a two stop difference in parts of theimage (1300), the camera's microprocessor accesses the database ofsimilar scenes (1310) and then selects the optimal metering for theimage and the user activates the digital sensor (1320), capturing animage. The camera provides the filtration to part of the image byadjusting the exposure in the processor (1330) and then stores the image(1340). In scenes such as sky on top, a neutral density filter is oftenrequired to remove the two stop difference between the bright top of theimage and the darker bottom of the image. By reducing the brightness atthe top, the exposure is evened out. By using the present system, thisprocess is accomplished in the camera.

In-camera special effects filtration is described in FIG. 14. After auser selects a special effects filtration technique (1400), the cameraassesses image meta-data (1410), analyzes image data (1420) and the usercaptures the image (1430). The camera's post-capture processor appliesspecial effects filtration technique to part of the image (1440) and thecamera stores the image in memory (1450).

Digital correction for sensor dust is described in FIG. 15. The patternof dust on the pixels of a digital sensor (1500) is assessed. Thespecific pixels that are obscured by the dust are identified (1510) anda map is built of a digital sensor pixel frame by comparing informationfrom at least two image samples to identify obscured pixels (1520) andthe camera takes an image (1530). Information from pixels that areadjacent to dust-affected pixels are used to interpolate data onaffected pixels (1540) and image data is processed and stored in memory(1550).

FIG. 16 shows an ASIC array in a digital imaging system. Once thedigital sensor (1620) captures an image, the digital file is sent tomultiple application specific integrated circuits (1630, 1640, 1650 and1660) for processing of several digital signals simultaneously. In oneembodiment of the invention, each ASIC corrects a single opticalaberration. In another embodiment, the ASICs will divide the digitalfile and perform specific functions on parts of the image file andreunite the file before storing the completed image in memory (1670).

In FIG. 17, the camera is illustrated with multiple interchangeabledigital sensors. The CCD (1720) is shown in the forward positioncapturing the digital image in this drawing. However, the CMOS (1730)digital imaging sensor may swivel around (1740) to replace the positionof the CCD in order to be used to capture the image. The image is thenprocessed (1750) and stored in memory (1760). The advantages ofinterchanging the digital sensors are to have the opportunity to benefitfrom the strengths of each sensor type without sacrificing theweaknesses of either. In some cases, the scene will be optimized for theexposure range of a CMOS digital sensor, while at other times, the scenewill be optimized for the detail of the CCD.

In FIG. 18, the nano-grid (1820) appears in front of the digital sensor(1830). The lens will send analog image data to the sensor through thenano-grid. Once the digital data from the sensor is processed (1840) itis sent to memory (1850). FIG. 19 shows a nano-grid (1900).Nanofilaments are shown at 1920. The modulation effects of thenanofilaments are shown at 1910. Nanofilaments will change theirposition in order to allow more or less light through the system.Nano-grids fit before, or in some cases, adjacent to and on top of thedigital sensor. Nano-grids behave as adjustable screens and may be usedto polarize light between the lens and the digital sensor. Nano-gridsare activated by electrical charge pulses sent to nanofilaments.

In FIG. 20, the feedback loop connecting the processor and the nano-gridis shown. Analog optical data passes through the nano-grid (2020) to thedigital sensor (2030) to the processor (DSP) (2040). A feedback loop(2050) is created by analyzing the data at the processor and modulatingthe performance of the nano-grid. For example, the image may be verybright initially, but the DSP will require that the image needs to bedarker in order to be properly exposed. In this case, it will activatethe nano-grid to modulate its grid structure in order to darken theimage so as to let less light through the filaments by adjusting thefilament structure. Once properly exposed, the camera will capture adigital image and send the file to memory (2060) for storage.

In FIG. 21, a tri-pixel architecture is described. The large (deeper)oval bucket in the center (2110) will capture highlight exposure, themedium sized oval bucket at the right (2120) will capture a normalexposure range and the small (shallower) oval bucket at the left (2130)will capture the shadow exposure range. These three separate wells arecontained within each pixel. In most cases, a pixel well is like abucket, capturing light data and passing the data sets to the digitalprocessor. By splitting the exposure range into three distinct wellswithin each pixel, the data is more accurately captured and organized inorder to accommodate the broader range of exposure data. This systemwill be able to increase the range of digital imaging up to four stops,from the typical two stops. By maximizing the range of exposure data,the image collection will display more tonal range that is moreassociated with films' broader exposure range.

FIG. 22 shows the pre-programmed module for in-camera functionality.After the camera detects the lens type and the sensor type (2200), thecamera focuses on an object (2210) and the processor accesses thelibrary of scene and filter types (2220). The user pre-selects specificin-camera modifications (2230) and activates the camera to capture animage (2240). The processor makes filtration adjustments after thesensor takes the image (2250) and the image file is stored (2260) inmemory.

Much like the nano-grid that is present before the digital image sensor,the pre-sensor modification to an image is shown in FIG. 23. An opticalfilter (2320) such as a polarizer filter is shown in a position betweenthe lens and the digital sensor (2340). The image is captured and theoptical filter is activated by the processor (2350), which determineswhether to activate the filter after analyzing a particular scene. Otherfiltration (2330), such as the use of the nano-grid, is alsointerchangeable with the optical filter in a position before the digitalsensor. In another embodiment of the present invention, the filterassembly is sequential, with both, or multiple, filters being activatedin order so as to obtain the maximum filtration effect. The DSP willcompute the optimal combination of filtration resources based on theavailable resources in the camera. In another embodiment, the opticalfilter assembly will fit out of the line of sight between the lens andthe digital sensor and will be activated by the DSP on-demand to swinginto the optical image path much like a mirror mechanism.

FIG. 24 shows the integration of optical and digital corrections withfeedback. Once the camera meters a scene (2400), the camera auto-focusesand obtains distance data to the object (2410), analyzes the lens data,the aperture data and the shutter speed data and accesses the library ofimages (2420). The camera assesses the optical aberrations (2430),activates the digital filters to correct aberrations (2440) and takesthe picture (2450). The camera analyzes the image and optimizes theimage using the processor (2460). This is performed by comparing theimage data to the library of images and assessing the needed filtrationrequired to optimize the image. The image is then stored in memory(2470).

The interactive feedback mechanism of integrated correction is describedin FIG. 25. After an analysis of the initial image (2500), theidentification of optical aberrations (2510) is made. The camera'sdatabase is accessed to retrieve the corrections (2520) and theinteractive feedback mechanism is activated (2530). Corrections tofilters are applied before the sensor captures the next image (2540). Inaddition, correction to the image signal via the DSP is made after thesensor data is captured (2550) and the image is stored in memory (2560).

The adaptive user pattern learning process is shown as images areprocessed in FIG. 26. A batch of images (2600, 2610, 2620 and 2630) isshown. The first image is processed (2640) and data from image 1 isanalyzed and metadata is mapped (2650). Image 2 is then analyzed and itsmetadata is compared to the image 1 metadata (2660). Similar metadata inimages 1 and 2 programmed in the processor (2670) and image 2 is rapidlyprocessed by using similar metadata from image 1 (2680). Images 3 and 4are analyzed and their metadata are compared to image 1 and rapidlyprocessed (2690).

In FIG. 27, the feedback process of filtering images is shown. After thecamera analyzes the user behaviors, user preferences and subjectbehaviors (2700), the camera's processor analyzes trends in the patternof user behavior and preferences and creates a map (2710). The camera'sprocessor anticipates the user behavior and subject behavior (2720) byapplying evolutionary computation and fuzzy logic algorithms. Thecamera's processor then applies a collaborative filtering process to anew image (2730) and the user takes and optimizes the image (2740).

FIG. 28 shows a software privacy function in a digital imaging system.The camera automatically downloads software as it enters a building orspecified area (2800). The camera's features are deactivated in specificlocations by the software (2810) while allowing the camera to performspecific camera functions by accessing software keys (2820). The camerathen performs specific functions (2830) that are available in therestricted area while other specific functions are disabled (2850).

In FIG. 29, the dynamics of zoom lens corrections are described. After alens detects a zoom lens type (2900), the camera detects a particularfocal length setting of the zoom lens (2910) and accesses a data libraryin the database to modify aberrations to optimize the zoom lens at aspecific focal setting (2920). The camera applies a correction to a lenssetting (2930) and continuously tracks the lens changes in the lensfocal length (2940). The camera then applies the changed corrections tothe changed focal lengths in the zoom lens (2950) and analyzes patternsof the user and subject behaviors (2960). Using fuzzy logic andevolutionary computation, the camera anticipates behaviors and rapidlyoptimizes image corrections by applying digital filtration (2970).

FIG. 30 shows object tracking in the dynamic changes of videocorrections. An object moves from position 1 (3010) to position 6 (3060)in sequence. The camera (3000) is stationary, yet it records theobject's motion as it is moving within the field of vision.

In FIG. 31, object tracking is described. Once the camera tracks anobject with auto-focus mechanisms and supplies distance information(3100), it tracks the user's zoom lens use patterns, anticipates thefocal length changes (3110) and, using evolutionary computation andfuzzy logic algorithms, predicts optimal exposures (3120) and capturesimages in real time, continuously tracking the object (3130).

FIG. 32 shows the caching process of a moving object in a stationaryscene. After the main objects are tracked by the camera (3200), thebackground of a stationary scene is cached in the camera (3210). Thecamera subtracts the data about objects from the background of the scene(3220). The background scene is then “blanked out” and cached in memory(3230) and the main objects are optimized by applying digital filters.

FIG. 33 describes the network coordination of a fixed sensor gridtracking multiple objects. The object in motion is represented inpositions A through D (3370, 3380, 3390 and 3395). The fixed sensors areat positions 1-6 (3300, 3310, 3320, 3330, 3340 and 3350). The image datais fed remotely into the central imager (3360). The sensors track theobject by taking image data at each fixed sensor position in thenetwork. As the object moves through the field of the network, eachsensor records the object.

In FIG. 34, wireless communication features of the digital imagingsystem are described. The camera (3400) sends digital image data fileswirelessly to a computer (3410). Files are also uploaded from the camerato the Internet in accessible locations (3450), while software files aredownloaded from the Internet to the camera, both automatically andthrough manual intervention. Files are automatically downloaded if theuser sets the camera to receive automated software updates. Oneadvantage of loading files automatically to the Internet is that theimages are then automatically published (3460) at a remote location, orlocations, for viewing.

FIG. 35 describes an image organization system. After images arecaptured by the camera (3500), the image files are organized accordingto metadata in an external computer database (3510). The image files arethen sorted by category and sent to the database of a distributednetwork (3520) for off-site storage.

Reference to the remaining portions of the specification, including thedrawings and claims, will realize other features and advantages of thepresent invention. Further features and advantages of the presentinvention, as well as the structure and operation of various embodimentsof the present invention, are described in detail below with respect toaccompanying drawings.

It is understood that the examples and embodiments described herein arefor illustrative purposes only and that various modifications or changesin light thereof will be suggested to persons skilled in the art and areto be included within the spirit and purview of this application andscope of the appended claims. All publications, patents, and patentapplications cited herein are hereby incorporated by reference for allpurposes in their entirety.

What is claimed is:
 1. A method of processing one or more images in adigital camera, comprising: electronically storing in the digital cameraat least one variable associated with capture of at least one image byan optical system of the digital camera, the optical system including adigital image sensor in the digital camera and a lens; digitallyprocessing the at least one image for image correction of the at leastone image, the digitally processing using upgradable in-camera softwarethat is configured to perform a plurality of image correction algorithmsand to access image correction data stored in an upgradable databasesystem, the in-camera software using the at least one variable; storingin memory one or more corrected images resulting from the digitallyprocessing the at least one image; detecting dust on the digital imagesensor; and digitally correcting for dust on the digital image sensor,wherein the in-camera software and database system are upgradable toprovide improved algorithms and correction data for correction ofimages.
 2. The method of claim 1, further, comprising: wherein the atleast one variable relates to prior in-camera settings for applicationby the in-camera software.
 3. The method of claim 1, wherein thedigitally processing the at least one image includes processingcorrections using at least one digital signal processor.
 4. The methodof claim 1, wherein the digitally processing the at least one imageincludes processing corrections using at least one application specificintegrated circuit.
 5. The method of claim 1, wherein the digitallyprocessing the at least one image includes processing corrections usingat least one microprocessor.
 6. The method of claim 1, wherein thedigitally processing the at least one image includes processing imageaberration corrections using hardware comprising at least one processorand at least one application specific circuit.
 7. The method of claim 1,wherein the digitally processing the at least one image includesprocessing image aberration corrections using combinations of processorsand application specific circuits.
 8. The method of claim 1, wherein theat least one variable includes an aperture setting.
 9. The method ofclaim 1, wherein the at least one variable includes lens data.
 10. Themethod of claim 1, wherein the at least one variable includes shutterspeed settings.
 11. The method of claim 1, wherein the at least onevariable includes ISO settings.
 12. The method of claim 1, wherein theat least one variable includes digital sensor data.
 13. The method ofclaim 1, wherein the at least one variable includes subject type. 14.The method of claim 1, wherein the digitally processing the at least oneimage for image correction further comprises adjusting a depth of fieldof an image.
 15. The method of claim 1, further comprising adjusting adepth of field of an image after capture of the image using thein-camera software.
 16. The method of claim 1, wherein the in-camerasoftware processes image corrections based on one or more prior imagecorrections.
 17. The method of claim 1, further comprising changing avariable based on an auto-focus process performed by the camera andelectronically storing that changed variable.
 18. The method of claim17, further comprising digitally processing the at least one image usingthe in-camera software to correct at least one optical image aberrationassociated with the optical system and the changed camera variable. 19.The method of claim 1, further comprising changing a variable based onan auto-exposure process performed by the camera and electronicallystoring that changed variable.
 20. The method of claim 19, furthercomprising digitally processing the at least one image using thein-camera software to correct at least one optical image aberrationassociated with the optical system and the changed camera variable. 21.The method of claim 1, further comprising changing a variable based on adepth of field adjustment process performed by the camera andelectronically storing that changed camera variable.
 22. The method ofclaim 21, further comprising digitally processing the at least one imageusing the in-camera software to correct at least one optical imageaberration associated with the optical system and the changed cameravariable.
 23. The method of claim 1, wherein the image processingalgorithms include a fast Fourier transform.
 24. The method of claim 1,further comprising: displaying a corrected image of the one or morecorrected images on a monitor, and based on a change to the at least onevariable, displaying a modified version of the corrected image.
 25. Themethod of claim 1, further comprising electronically storing updatedin-camera software.
 26. The method of claim 25, further comprisingupdating the image correction data with lens correction data.
 27. Themethod of claim 25, comprising updating in-camera software usingwireless communications.
 28. The method of claim 25, comprising updatingin-camera software using the INTERNET.
 29. The method of claim 1,further comprising: correcting for a dead pixel of the digital imagesensor.
 30. The method of claim 1, further comprising: digitallyprocessing a plurality of images to correct video taken by the digitalcamera; and storing a plurality of corrected images.